Method and apparatus with adaptive super sampling

1 . An adaptive super sampling method, the method comprising:

rendering frames of a three-dimensional (3D) object, the frames comprising a current frame and a previous frame preceding the current frame;

determining motion vectors indicating a correspondence relationship between pixels in the current frame and pixels in the previous frame;

generating first and second geometric identifier maps (G-ID maps) respectively corresponding to the current frame and the previous frame based on geometric elements of the 3D object that are associated with the pixels in the current frame and the previous frame, each G-ID map comprising identifiers of the geometric elements of the 3D object;

based on the motion vectors and the first and second G-ID maps, generating an artifact map predicting where artifacts will occur from inter-frame super sampling of the current frame; and

performing adaptive super sampling on the current frame based on the artifact map.

2 . The method of claim 1 , wherein the generating of the first and second G-ID maps comprises:

determining associations between the geometrical elements and the pixels in the current frame and the previous frame;

generating G-IDs for the pixels by based on the 3D-geometrical elements; and

generating the first and second G-ID maps by encoding the G-IDs of the pixels of the current and previous frame.

3 . The method of claim 2 , wherein the identifiers of the geometrical elements comprises any one or any combination of:

identification information of at least one object comprised in the current frame and the previous frames, identification information of polygons constituting the at least one object, identification information of vertexes comprised in the polygons, identification information of edges comprised in the polygons, subdivision levels of the polygons, and identification information of sub-polygons constituting the polygons.

4 . The method of claim 1 , wherein the generating of the artifact map comprises:

matching pixels corresponding to a same position in the current frame and the previous frame, using the motion vectors; and

generating the artifact map based on whether G-IDs of matched pixels in the first and second G-ID maps are the same.

5 . The method of claim 4 , wherein the matching of the pixels comprises:

matching second pixels of the previous frame to third pixels at positions corresponding to the second pixels in a third frame obtained by warping the current frame to the previous frame using the motion vectors.

6 . The method of claim 4 , wherein the generating of the artifact map comprises:

assigning a first value to elements in the artifact map corresponding to first regions in which matched pixels have a same G-ID; and

assigning a second value to elements in the artifact map corresponding to second regions in which matched pixels have different G-IDs.

7 . The method of claim 1 , wherein the performing of the adaptive super sampling comprises:

according to the artifact map, performing inter-frame super sampling on some regions of the current frame and performing intra-frame super sampling on other regions of the current frame, wherein the artifact map controls which regions are inter-frame super sampled and which regions are intra-frame super sampled.

8 . The method of claim 1 , wherein the performing of the adaptive super sampling comprises:

up-sampling, using pixel values of the previous frame, regions of the current frame corresponding to elements of the artifact map having a first value; and

performing only single image super sampling on regions of the current frame corresponding to elements of the artifact map having a second value.

9 . The method of claim 8 , wherein the up-sampling is performed by neural super sampling using the pixel values of the previous frame obtained based on the motion vectors.

10 . The method of claim 1 , wherein the first and second G-ID maps are generated based on any combination of: an object comprised in the 3D object, polygons constituting the object in the 3D object, and vertexes and edges of the polygons.

11 . An adaptive super sampling apparatus, the apparatus comprising:

one or more processors;

memory storing instructions configured to cause the one or more processors to:

render frames of a 3D medel object including a current frame and a previous frame;

to determine motion vectors indicating a correspondence relationship between pixels comprised in the current frame and the previous frame;

generate a first G-ID map and a second G-ID map respectively corresponding to the current frame and the previous frame based on geometric properties elements of the 3D object associated with the pixels in the current frame and the previous frame, each G-ID map comprising identifiers of the geometric elements of the 3D object;

based on the motion vectors and the first and second G-ID maps, generate an artifact map indicating where artifacts will occur during super sampling of the current frame; and

perform adaptive super sampling on the current frame based on the artifact map.

12 . The apparatus of claim 11 , wherein the instructions are further configured to cause the one or more processors to:

generate the first and second G-ID maps based on which of the geometric elements of the 3D object are determined to be associated with which of the pixels in the current and previous frame.

13 . The apparatus of claim 12 , wherein the identifiers of the geometric elements comprise any one or any combination of:

information of an object comprised in the current frame and the previous frame, information of polygons constituting the object, information of vertexes comprised in the polygons, information of edges comprised in the polygons, subdivision levels of the polygons, and information of sub-polygons constituting the polygons.

14 . The apparatus of claim 11 , wherein the instructions are further configured to cause the one or more processors to:

match pixels corresponding to a same position in the current frame and the previous frame, using the motion vectors; and

generate the artifact map based on whether G-IDs of the matched pixels in the first and second G-ID maps are the same.

15 . The apparatus of claim 11 , wherein the instructions are further configured to cause the one or more processors to:

assign a first value to elements of the artifact map corresponding to matched pixels, in response to each of the matched pixels having the same G-ID; and

assign a second value to elements of the artifact map corresponding to matched pixels having different G-IDs.

16 . The apparatus of claim 15 , wherein the instructions are further configured to cause the one or more processors to:

perform temporal inter-frame super sampling on only regions of the current frame corresponding elements in the artifact map having the first value.

17 . The apparatus of claim 16 , wherein the instructions are further configured to cause the one or more processors to:

perform intra-frame up-sampling for only regions of the current frame corresponding elements of the artifact map having the second value.

18 . The apparatus of claim 16 , wherein the instructions are further configured to cause the one or more processors to:

perform the temporal inter-frame super sampling by applying neural super sampling to pixel values of the previous frame.

19 . A method comprising:

receiving a current frame and a previous frame rendered from a 3D object comprising data;

generating motion vectors based on the current frame and the previous frame;

determining first associations between pixels of the current frame and identifiers of 3D geometric elements of the 3D object;

determining second associations between pixels of the previous frame and the identifiers of the geometric elements of the 3D object;

determining which regions of the current frame to perform inter-frame super sampling on based on comparing the first associations with the second associations.

20 . The method of claim 19 , wherein one of the regions is determined based on corresponding pixels of the current and previous frame being determined to be associated with a same geometric element identifier.