Method and apparatus with supersampling

1 . A supersampling method, the method comprising:

receiving a low-resolution image comprising a current frame;

generating a first low-resolution partial image by sampling first sub-pixel regions of the current frame;

warping a high-resolution image, of a previous frame preceding the current frame, which has been outputted from a neural network, to a current view corresponding to the current frame;

replacing a first partial region of the warped high-resolution image of the previous frame with image data from the first low-resolution partial image;

generating a second low-resolution partial image by sampling second sub-pixel regions of the current frame after sampling the first sub-pixel regions;

warping the high-resolution image, of the previous frame preceding the current frame, which has been outputted from the neural network and to which the first partial region has been replaced with the image data from the first low-resolution partial image, to the current view corresponding to the current frame;

replacing a second partial region of the high-resolution image, to which the first partial region has been replaced with the image data from the first low-resolution partial image, of the previous frame, with image data from the second low-resolution partial image; and

generating a high-resolution image of the current frame by applying the high-resolution image of the previous frame, in which at least the first and second partial regions have been replaced, to the neural network.

2 . The method of claim 1 , wherein the generating of the first low-resolution partial image comprises:

determining the first sub-pixel regions of one pixel region of the current frame by dividing the one pixel region into one high-resolution pixel region that is to be up-scaled; and

generating the first low-resolution partial image corresponding to each of the first sub-pixel regions by sampling each of the first sub-pixel regions at regular periods.

3 . The method of claim 1 , wherein the replacing the first partial region is based on a disocclusion map.

4 . The method of claim 1 , wherein the replacing the first partial region comprises:

dividing the warped high-resolution image of the previous frame into a set of low-resolution partial images of respectively corresponding channels; and

replacing one of the divided partial images with the generated first low-resolution partial image.

5 . The method of claim 1 , wherein:

the warping is based on a low-resolution motion vector map corresponding to the current frame and the previous frame.

6 . The method of claim 5 , wherein the low-resolution motion vector map indicates movement relationships between first pixels of the current frame and second pixels of the previous frame, and wherein the method further comprises generating the low-resolution motion vector map by rendering the low-resolution image.

7 . The method of claim 5 , further comprising:

up-scaling the low-resolution motion vector map to a high-resolution scaled motion vector map.

8 . The method of claim 7 , wherein the up-scaling is performed using an upscaling filter comprising a nearest neighborhood filter, a bilinear filter, or a bicubic filter.

9 . The method of claim 7 , wherein the warping comprises:

warping the high-resolution image of the previous frame output from the neural network to the current view based on the high-resolution scaled motion vector map.

10 . The method of claim 1 , further comprising:

comparing the previous frame with the current frame; and

based on the comparing, generating a disocclusion map that identifies disocclusion regions of the current frame that were not visible in the previous frame and that are visible in the current frame.

11 . The method of claim 10 , wherein the disocclusion regions are identified by instances of a first value in the disocclusion map and non-disocclusion regions are identified by instances of a second value in the disocclusion map.

12 . The method of claim 10 , further comprising:

masking the warped high-resolution image of the previous frame using the disocclusion map.

13 . The method of claim 12 , wherein the masking comprises:

applying the disocclusion map to the warped high-resolution image of the previous frame such that:

a region of the warped high-resolution image corresponding to a disocclusion region of the disocclusion map is cleared; and

a region of the warped high-resolution image corresponding to a non-occlusion region of the disocclusion map retains its original pixel values.

14 . The method of claim 10 , wherein the replacing the first partial region comprises:

replacing a partial region of the high-resolution image of the previous frame masked by the disocclusion map with the image data from the first low-resolution partial image.

15 . The method of claim 1 , wherein the neural network has an input resolution and an output resolution that are the same.

16 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

17 . A supersampling apparatus, the apparatus comprising:

storage storing a low-resolution image comprising a current frame and storing a previous frame preceding the current frame; and

one or more processors configured to:

generate a first low-resolution partial image by sampling first sub-pixel regions of the current frame;

warp a high-resolution image, of a previous frame preceding the current frame, which has been outputted from a neural network, to a current view corresponding to the current frame;

replace a first partial region of the warped high-resolution image of the previous frame with image data from the first low-resolution partial image;

generate a second low-resolution partial image by sampling second sub-pixel regions of the current frame after sampling the first sub-pixel regions;

warp the high-resolution image, of the previous frame preceding the current frame, which has been outputted from the neural network and to which the first partial region has been replaced with the image data from the first low-resolution partial image, to the current view corresponding to the current frame;

replace a second partial region of the high-resolution image, to which the first partial region has been replaced with the image data from the first low-resolution partial image, of the previous frame, with image data from the second low-resolution partial image; and

generate a high-resolution image of the current frame by applying the high-resolution image of the previous frame, in which at least the first and second partial regions have been replaced, to the neural network.

18 . The apparatus of claim 17 , wherein the one or more processors are further configured to:

determine the first sub-pixel regions of the current frame by dividing one pixel region of the current frame into one high-resolution pixel region to be up-scaled and generate the first low-resolution partial image by sampling each of the first sub-pixel regions.

19 . The apparatus of claim 17 , further comprising:

a memory comprising the neural network, wherein the neural network has been trained to output the high-resolution image of the current frame in response to the high-resolution image of the previous frame being input.