Image processing apparatus and method of processing multi-frames using the same
An image processing apparatus, including a memory configured to store one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to cause the image processing apparatus to: identify, in a previous frame, a prediction sample corresponding to a current sample of a current frame, generate a prediction frame for the current frame by changing a sample value of a collocated sample of the previous frame according to a sample value of the prediction sample, derive a weight by comparing a sample value of the current sample with the sample value of the prediction sample, apply the weight to a collocated sample of the prediction frame to obtain a weighted prediction frame, and obtain a current output frame by processing the current frame and the weighted prediction frame through a neural network comprising a convolution layer.
1. An image processing apparatus comprising:
a memory configured to store one or more instructions; and
a processor configured to execute the one or more instructions stored in the memory to cause the image processing apparatus to:
identify, in a previous frame, a prediction sample corresponding to a current sample of a current frame,
generate a prediction frame for the current frame by using the prediction sample,
generate a prediction output frame from a previous output frame corresponding to the previous frame according to a positional relationship between the current sample and the prediction sample,
derive a weight by comparing a sample value of the current sample with the sample value of the prediction sample,
apply the weight to collocated samples of the prediction frame and the prediction output frame, to obtain a weighted prediction frame and a weighted prediction output frame, and
obtain a current output frame by processing the current frame, the weighted prediction frame and the weighted prediction output frame through a neural network comprising a convolution layer,
wherein the collocated samples of the prediction frame and the prediction output frame are collocated with the current sample.
2. The image processing apparatus of claim 1 , wherein the processor is further configured to cause the image processing apparatus to identify, as the prediction sample, a most similar sample of the previous frame, wherein the most similar sample has a sample value that is most similar to the sample value of the current sample from among sample values of a collocated sample of the previous frame and neighboring samples of the collocated sample of the previous frame,
wherein the collocated sample of the previous frame is collocated with the current sample.
3. The image processing apparatus of claim 2 , wherein the processor is further configured to cause the image processing apparatus to:
perform convolution processing on the current sample and neighboring samples of the current sample by using a pre-determined first filter kernel to obtain a first sample value corresponding to the first filter kernel,
perform the convolution processing on the collocated sample of the previous frame and the neighboring samples of the collocated sample by using a plurality of pre-determined second filter kernels to obtain second sample values corresponding to the plurality of second filter kernels,
identify a second sample value that is most similar to the first sample value from among the second sample values, and
determine, as the prediction sample, a corresponding sample which corresponds to the identified second sample value from among the collocated sample of the previous frame and the neighboring samples of the collocated sample of the previous frame.
4. The image processing apparatus of claim 3 , wherein, in the first filter kernel, a first filter sample corresponding to the current sample has a pre-set first value, and remaining first filter samples have a value of 0.
5. The image processing apparatus of claim 4 , wherein, in the plurality of second filter kernels, one second filter sample has a pre-set second value, and remaining second filter samples have the value of 0,
wherein a location of the one second filter sample is different for each of the plurality of second filter kernels.
6. The image processing apparatus of claim 5 , wherein a sign of the pre-set first value is opposite to a sign of the pre-set second value-.
7. The image processing apparatus of claim 3 , wherein the processor is further configured to cause the image processing apparatus to execute the one of perform the convolution processing on the collocated sample of the previous frame and the neighboring samples of the collocated sample of the previous frame by using a pre-determined third filter kernel to generate the prediction frame,
wherein, in the third filter kernel, a third filter sample corresponding to the prediction sample has a pre-set third value, and remaining third filter samples have a value of 0.
8. The image processing apparatus of claim 1 , wherein the weight is inversely proportional to a difference between the sample value of the current sample and the sample value of the prediction sample.
9. The image processing apparatus of claim 1 , wherein the processor is further configured to cause the image processing apparatus to:
obtain a previous feature map output as a result of processing the previous frame by using the neural network,
generate a prediction feature map from the previous feature map according to the positional relationship between the current sample and the prediction sample,
apply the weight to a collocated samples of the prediction feature map to obtain a weighted prediction feature map, and
input the weighted prediction output frame, the weighted prediction feature map, the weighted prediction frame, and the current frame, to the neural network.
10. The image processing apparatus of claim 9 , wherein the neural network comprises a plurality of sub-neural networks comprising a first convolution layer, a second convolution layer, and a plurality of third convolution layers,
wherein the first convolution layer of a first sub-neural network is configured to perform convolution processing on a result of concatenating the weighted prediction output frame, the weighted prediction frame, and the current frame,
wherein the second convolution layer of the first sub-neural network is configured to perform the convolution processing on the weighted prediction feature map, and
wherein the plurality of third convolution layers of the first sub-neural network are configured to sequentially perform the convolution processing on a result of concatenating a feature map output from the first convolution layer of the first sub-neural network and a feature map output from the second convolution layer of the first sub-neural network.
11. The image processing apparatus of claim 10 , wherein the first convolution layer of a sub-neural network other than the first sub-neural network is configured to perform the convolution processing on a result of concatenating the weighted prediction frame, the current frame, and an intermediate output frame output from a previous sub-neural network,
wherein the second convolution layer of the sub-neural network other than the first sub-neural network is configured to perform the convolution processing on an intermediate feature map output from the previous sub-neural network, and
wherein the plurality of third convolution layers of the sub-neural network other than the first sub-neural network are configured to sequentially perform the convolution processing on a result of concatenating a feature map output from the first convolution layer of the sub-neural network other than the first sub-neural network and a feature map output from the second convolution layer of the sub-neural network other than the first sub-neural network.
12. The image processing apparatus of claim 1 , wherein the previous output frame comprises a first previous output frame output from the neural network, and a second previous output frame obtained as a result of processing the first previous output frame, which is reconstructed through encoding and decoding the first previous output frame, through the neural network.
13. The image processing apparatus of claim 1 , wherein the processor is further configured to cause the image processing apparatus to transmit, to a terminal device, a bitstream generated through encoding performed on the current output frame.
14. A method, performed by an image processing apparatus, of processing multi-frames, the method comprising:
identifying, in a previous frame, a prediction sample corresponding to a current sample of a current frame;
generating a prediction frame for the current frame by using the prediction sample;
generating a prediction output frame from a previous output frame corresponding to the previous frame according to a positional relationship between the current sample and the prediction sample;
deriving a weight by comparing a sample value of the current sample with the sample value of the prediction sample;
applying the weight to collocated samples of the prediction frame and the prediction output frame, to obtain a weighted prediction frame and a weighted prediction output frame; and
processing the current frame, the weighted prediction frame and the weighted prediction output frame through a neural network comprising a convolution layer to obtain a current output frame,
wherein the collocated samples of the prediction frame and the prediction output frame are collocated with the current sample.
15. A non-transitory computer-readable recording medium storing a program which, when executed by at least one processor, causes an image processing apparatus to:
identifying, in a previous frame, a prediction sample corresponding to a current sample of a current frame;
generating a prediction frame for the current frame by using the prediction sample;
generating a prediction output frame from a previous output frame corresponding to the previous frame according to a positional relationship between the current sample and the prediction sample;
deriving a weight by comparing a sample value of the current sample with the sample value of the prediction sample;
applying the weight to collocated samples of the prediction frame and the prediction output frame, to obtain a weighted prediction frame and a weighted prediction output frame; and
processing the current frame, the weighted prediction frame and the weighted prediction output frame through a neural network comprising a convolution layer to obtain a current output frame,
wherein the collocated samples of the prediction frame and the prediction output frame are collocated with the current sample.