Apparatus and method for applying artificial intelligence-based filtering to image
An image processing apparatus including a memory storing instructions, and a processor configured to operate based on the instructions to reconstruct a current block from encoded data of the current block, obtain a quantization error map including sample values calculated based on a quantization parameter included in the encoded data, obtain a first modified block by applying the current block and the quantization error map to a neural network, obtain a first differential block between the current block and the first modified block, obtain a second differential block by changing sample values of the first differential block based on a parameter dependent on a feature of the current block, and obtain a second modified block by combining the current block and the second differential block.
1. An image processing apparatus comprising:
a memory storing one or more instructions; and
a processor configured to operate based on the one or more instructions to:
reconstruct a current block from encoded data of the current block;
obtain a quantization error map comprising sample values calculated based on a quantization parameter;
calculate an extension distance based on at least one among a number of convolutional layers in a neural network and a size of a filter kernel used by the convolutional layers;
obtain a first modified block by applying, to the neural network, an extended block and the quantization error map, wherein the extended block comprises samples of the current block and neighboring samples corresponding to the extension distance from among neighboring samples outside a boundary of the current block in a current image; and
obtain a second modified block from the first modified block based on a parameter dependent on a feature of the current block.
2. The image processing apparatus of claim 1 , wherein the processor is further configured to:
obtain a first differential block between the current block and the first modified block;
obtain a second differential block by changing sample values of the first differential block, the changing of the sample values being based on the parameter dependent on the feature of the current block; and
obtain the second modified block by combining the current block and the second differential block.
3. The image processing apparatus of claim 1 , wherein the sample values of the quantization error map correspond to a quantization step size or a quantization error value calculated based on the quantization parameter.
4. The image processing apparatus of claim 1 , wherein, when quantization parameters for lower blocks of the current block are included in the encoded data, the sample values of the quantization error map are calculated for each of the lower blocks.
5. The image processing apparatus of claim 2 , wherein the processor is further configured to:
select a parameter indicated by first information obtained from the encoded data, from among a plurality of candidate parameters; and
obtain the second differential block by changing the sample values of the first differential block based on the selected parameter.
6. The image processing apparatus of claim 5 , wherein the processor is further configured to:
select a parameter set indicated by second information obtained from the encoded data, from among a plurality of parameter sets; and
select the parameter indicated by the first information, from among the plurality of candidate parameters in the selected parameter set.
7. The image processing apparatus of claim 6 , wherein the second information is obtained for an upper block comprising the current block, and
wherein the first information is obtained for the current block.
8. The image processing apparatus of claim 1 , wherein the processor is further configured to obtain a feature value of the current block or an upper block of the current block to determine the parameter.
9. The image processing apparatus of claim 8 , wherein the feature value is obtained based on at least one of i) a sum of squares of residual sample values obtained to reconstruct the current block or the upper block, ii) an average of the squares of the residual sample values, iii) a maximum value from among the residual sample values, iv) a maximum value from among absolute values of the residual sample values, v) a number of non-zero (0) transformation coefficients from among transformation coefficients corresponding to residual samples, vi) a value indicating a type of a slice corresponding to the current block, vii) a ratio between an area of a block to which intra prediction is applied and an area of a block to which inter prediction is applied, in the current block or the upper block, viii) a sharpness of the current block or the upper block, ix) an average of one or more quantization step sizes calculated based on one or more quantization parameters set for the current block or the upper block, or x) an average of one or more quantization error values calculated from the one or more quantization parameters.
10. The image processing apparatus of claim 2 , wherein the parameter comprises a scale factor, and
wherein the processor is further configured to obtain the second differential block by scaling the sample values of the first differential block, the scaling being based on the scale factor.
11. The image processing apparatus of claim 2 , wherein the parameter comprises a clipping factor, and
wherein the processor is further configured to obtain the second differential block by clipping the sample values of the first differential block based on an upper limit and a lower limit corresponding to the clipping factor.
12. The image processing apparatus of claim 1 , wherein the processor is further configured to:
obtain a feature value of the current block or an upper block of the current block; and
select a weight set corresponding to the feature value from among a plurality of weight sets, and
wherein the extended block and the quantization error map are processed by a neural network operating based on the selected weight set.
13. The image processing apparatus of claim 1 , wherein the processor is further configured to:
obtain a feature value of the current block or an upper block of the current block; and
determine whether to apply an AI-based filtering to the current block, based on the obtained feature value.
14. The image processing apparatus of claim 1 , the neural network comprises:
at least one convolutional layer configured to output a first differential block by performing convolution on the extended block and the quantization error map, the performing of the convolution being based on a preset weight; and
a summation layer configured to output the first modified block by adding the first differential block and the current block.
15. The image processing apparatus of claim 14 , wherein, when the boundary of the current block corresponds to a boundary of the current image, the neighboring samples corresponding to the extension distance are determined from available closest samples.
16. The image processing apparatus of claim 1 , wherein the neural network is trained based on loss information corresponding to a difference between an original block for training, and a first modified block for training, which is obtained through the neural network, and
wherein the first modified block for training is obtained by applying, to the neural network, a current block for training, which is obtained by encoding and decoding the original block for training, and a quantization error map for training, which corresponds to the current block for training.
17. An image processing method comprising:
reconstructing a current block from encoded data of the current block;
obtaining a quantization error map comprising sample values calculated based on a quantization parameter;
calculating an extension distance based on at least one among a number of convolutional layers in a neural network and a size of a filter kernel used by the convolutional layers;
obtaining a first modified block by applying, to the neural network, an extended block and the quantization error map, wherein the extended block comprises samples of the current block and neighboring samples corresponding to the extension distance from among neighboring samples outside a boundary of the current block in a current image and
obtaining a second modified block from the first modified block based on a parameter dependent on a feature of the current block.
18. A non-transitory computer-readable recording medium having recorded thereon a program for executing the method of claim 17 on a computer.
19. An image processing apparatus comprising:
a memory storing one or more instructions; and
a processor configured to operate based on the one or more instructions to:
reconstruct a current block from encoded data of the current block;
obtain a quantization error map comprising sample values calculated based on a quantization parameter;
calculate an extension distance based on at least one among a number of convolutional layers in a neural network and a size of a filter kernel used by the convolutional layers;
obtain a first differential block by applying, to the neural network, an extended block and the quantization error map, wherein the extended block comprises samples of the current block and neighboring samples corresponding to the extension distance from among neighboring samples outside a boundary of the current block in a current image;
obtain a second differential block by changing sample values of the first differential block, the changing of the sample values being based on a parameter dependent on a feature of the current block; and
obtain a modified block by combining the current block and the second differential block.