Image processing device and operation method thereof
An image processing apparatus includes a memory storing at least one instruction; and a processor configured to execute the at least one instruction to use at least one neural network to: extract n pieces of first feature information from a first image, based on locations of pixels included in the first image, wherein n is a positive integer, generate n pieces of second feature information by performing a convolution operation between each of the n pieces of the first feature information and each of n kernels, and generate, based on the n pieces of the second feature information, a second image from which compression artifacts included in the first image are removed.
1 . An image processing apparatus comprising:
memory storing instructions; and
at least one processor,
wherein the instructions, when executed by the at least one processor, cause the image processing apparatus to use at least one neural network to:
extract n pieces of first feature information from a first image, based on locations of pixels included in the first image, wherein n is a positive integer, by:
dividing, prior to performing a convolution operation on the first image, the first image into a first plurality of blocks having a variable size, that is preset before extraction, based on the following equation:
F
in
u
,
v
=
F
in
[
u
:
L
:
W
-
1
:
v
:
L
:
H
-
1
,
0
:
1
:
C
-
1
]
where
F
in
u
,
v
denotes a sampling feature, F in denotes the first image having a width W, a height H, and C channels, u:L:W−1 denotes sampling of pixels with a preset sampling period L in a width direction from position u to position W−1, v:L:H−1 denotes sampling of pixels with the preset sampling period L in a height direction from position v to position H−1, and 0:1:C−1 denotes sampling of all pixels across the C channels, such that a plurality of corresponding sampling pixels have a same location among the first plurality of blocks, the plurality of corresponding sampling pixels have different locations in the first image, and the plurality of corresponding sampling pixels are assigned a same group from among n groups, and
generating the n pieces of first feature information such that the n pieces of first feature information correspond to the n groups,
based on the n pieces of first feature information being extracted from the first image, generate n pieces of second feature information by performing a convolution operation between each of the n pieces of first feature information and each of n kernels, and
generate, based on the n pieces of second feature information, a second image from which compression artifacts included in the first image are removed.
2 . The image processing apparatus of claim 1 , wherein each of the first plurality of blocks comprises n pixels, and
wherein n is determined based on the variable size that is preset before extraction.
3 . The image processing apparatus of claim 1 , wherein the instructions, when executed by the at least one processor, cause the image processing apparatus to use the at least one neural network to:
divide the second image into a second plurality of blocks having the variable size that is preset before extraction, and
determine pixel values included in one piece of second feature information, among the n pieces of second feature information, as pixel values of a same location in the second plurality of blocks.
4 . The image processing apparatus of claim 1 , wherein the instructions, when executed by the at least one processor, cause the image processing apparatus to use the at least one neural network to extract the n pieces of first feature information, by performing a first convolution operation between the first image and a second kernel, by applying a stride size.
5 . The image processing apparatus of claim 4 , wherein n is determined based on the stride size.
6 . The image processing apparatus of claim 1 , wherein the n kernels are different from each other.
7 . The image processing apparatus of claim 1 , wherein a size of the first image and a size of the second image are the same.
8 . An operating method of an image processing apparatus for performing image processing by using at least one neural network, the operating method comprising:
extracting n pieces of first feature information from a first image, based on locations of pixels included in the first image, wherein n is a positive integer, by:
dividing, prior to performing a convolution operation on the first image, the first image into a first plurality of blocks having a variable size, that is preset before extraction, based on the following equation:
F
in
u
,
v
=
F
in
[
u
:
L
:
W
-
1
:
v
:
L
:
H
-
1
,
0
:
1
:
C
-
1
]
where
F
in
u
,
v
denotes a sampling feature, Fin denotes the first image having a width W, a height H, and C channels, u:L:W−1 denotes sampling of pixels with a preset sampling period L in a width direction from position u to position W−1, v:L:H−1 denotes sampling of pixels with the preset sampling period L in a height direction from position v to position H−1, and 0:1:C−1 denotes sampling of all pixels across the C channels, such that a plurality of corresponding sampling pixels have a same location among the first plurality of blocks, the plurality of corresponding sampling pixels have different locations in the first image, and the plurality of corresponding sampling pixels are assigned a same group from among n groups; and
generating the n pieces of first feature information such that the n pieces of first feature information correspond to the n groups;
based on the n pieces of first feature information being extracted from the first image, generating n pieces of second feature information, by performing a convolution operation between each of the n pieces of first feature information and each of n kernels; and
generating, based on the n pieces of second feature information, a second image from which compression artifacts included in the first image are removed.
9 . The operating method of claim 8 , wherein each of the first plurality of blocks comprises n pixels, and
wherein n is determined based on the variable size that is preset before extraction.
10 . The operating method of claim 8 , wherein the generating the second image comprises:
dividing the second image into a second plurality of blocks having the variable size that is preset before extraction; and
determining pixels included in one piece of second feature information, among the n pieces of second feature information, as pixel values of a same location in the second plurality of blocks.
11 . The operating method of claim 8 , wherein the extracting the n pieces of first feature information comprises extracting the n pieces of first feature information, by performing a first convolution operation between the first image and a second kernel, by applying a stride size.
12 . The operating method of claim 11 , wherein n is determined based on the stride size.
13 . A computer program product comprising a non-transitory computer-readable recording medium having recorded thereon a program for executing an operating method of an image processing apparatus for performing image processing by using at least one neural network, the operating method comprising:
extracting n pieces of first feature information from a first image, based on locations of pixels included in the first image, wherein n is a positive integer, by:
dividing, prior to performing a convolution operation on the first image, the first image into a first plurality of blocks having a variable size, that is preset before extraction, based on the following equation:
F
in
u
,
v
=
F
in
[
u
:
L
:
W
-
1
:
v
:
L
:
H
-
1
,
0
:
1
:
C
-
1
]
where
F
in
u
,
v
in denotes a sampling feature, F in denotes the first image having a width W, a height H, and C channels, u:L:W−1 denotes sampling of pixels with a preset sampling period L in a width direction from position u to position W−1, v:L:H−1 denotes sampling of pixels with the preset sampling period L in a height direction from position v to position H−1, and 0:1:C−1 denotes sampling of all pixels across the C channels, such that a plurality of corresponding sampling pixels have a same location among the first plurality of blocks, the plurality of corresponding sampling pixels have different locations in the first image, and the plurality of corresponding sampling pixels are assigned a same group from among n groups; and
generating the n pieces of first feature information such that the n pieces of first feature information correspond to the n groups;
based on the n pieces of first feature information being extracted from the first image, generating n pieces of second feature information, by performing a convolution operation between each of the n pieces of first feature information and each of n kernels; and
generating, based on the n pieces of second feature information, a second image from which compression artifacts included in the first image are removed.
14 . The computer program product of claim 13 , wherein each of the first plurality of blocks comprises n pixels, and
wherein n is determined based on the variable size that is preset before extraction.
15 . The computer program product of claim 13 , wherein the generating the second image comprises:
dividing the second image into a second plurality of blocks having the variable size that is preset before extraction; and
determining pixels included in one piece of second feature information, among the n pieces of second feature information, as pixel values of a same location in the second plurality of blocks.
16 . The computer program product of claim 13 , wherein the extracting the n pieces of first feature information comprises extracting the n pieces of first feature information, by performing a first convolution operation between the first image and a second kernel, by applying a stride size.
17 . The computer program product of claim 16 , wherein n is determined based on the stride size.
18 . The image processing apparatus of claim 1 , wherein the variable size that is preset before extraction is determined based on (i) the sampling period in the width direction and the height direction and (ii) at least one of: whether line-based processing is applied to the first image, whether de-interlacing is applied to the first image, or compression parameters encoded in the first image.