Method and device with image generation based on neural scene representation

1 . A processor-implemented method, comprising:

extracting pyramid level color feature maps, each being extracted from two or more images;

generating a cost volume of a corresponding pyramid level through cost matching between color feature maps of the same pyramid level among the color feature maps;

extracting pyramid level density feature maps based on the cost volume;

generating neural scene representation (NSR) cube information representing a three-dimensional (3D) space based on the color feature maps and the density feature maps; and

generating a two-dimensional (2D) scene of a field of view (FOV) different from a FOV of the two or more images based on the NSR cube information.

2 . The method of claim 1 , wherein the two or more images are images captured by cameras positioned within a critical distance from each other.

3 . The method of claim 1 , wherein the two or more images are images captured by cameras having optical axes parallel to each other.

4 . The method of claim 1 , wherein the extracting of the color feature maps comprises:

performing feature extraction on a first image among the two or more images, based on data generated by feature extraction performed on a second image among the two or more images; and

performing feature extraction on the second image based on the data generated by the feature extraction on the first image.

5 . The method of claim 1 , wherein the extracting of the color feature maps comprises:

refining a feature map of a layer of a first feature extraction network for a first image using a feature map of a layer of a second feature extraction network for a second image; and

generating a feature map to be propagated to a subsequent layer of the first feature extraction network.

6 . The method of claim 5 , comprising:

generating an epipolar attention map based on the feature map of the layer of the first feature extraction network and the feature map of the layer of the second feature extraction network; and

generating feature maps for the subsequent layer of the first feature extraction network and a subsequent layer of the second feature extraction network based on the epipolar attention map.

7 . The method of claim 1 , wherein the extracting of the color feature maps comprises:

performing epipolar attention on three or more images comprising the two or more images by performing feature extraction on a first image among the three or more images based on a result of feature extraction performed on a second image among the three or more images and feature extraction performed on a third image among the three or more images.

8 . The method of claim 1 , wherein the extracting of the density feature maps comprises:

generating a cost volume based on a correlation for each pyramid level between color feature maps extracted from a first image and color feature maps extracted from a second image; and

generating a density feature map for a corresponding pyramid level based on the cost volume.

9 . The method of claim 1 , wherein the generating of the NSR cube information comprises:

generating the NSR cube information by storing, in each position in the 3D space, an NSR statistical value of color feature maps and density feature maps extracted from images of a plurality of viewpoints in the same 3D space.

10 . The method of claim 1 , wherein the generating of the 2D scene comprises:

determining an NSR parameter from the NSR cube information for each of positions in the 3D space along a view direction from a pixel of the 2D scene;

determining a pixel value of the pixel by performing volume rendering based on NSR parameters of the positions in the 3D space along the view direction; and

reconstructing the 2D scene by performing volume rendering on pixels of the 2D scene.

11 . An electronic device, comprising:

one or more processors configured to:

extract pyramid level color feature maps, each being extracted from two or more images;

generating a cost volume of a corresponding pyramid level through cost matching between color feature maps of the same pyramid level among the color feature maps;

extract pyramid level density feature maps based on the cost volume;

generate neural scene representation (NSR) cube information representing a three-dimensional (3D) space based on the color feature maps and the density feature maps; and

generate a two-dimensional (2D) scene of a field of view (FOV) different from a FOV of the two or more images based on the NSR cube information.

12 . The electronic device of claim 11 , wherein the two or more images are images captured by either one or both of:

cameras positioned within a critical distance from each other; and

cameras having optical axes parallel to each other.

13 . The electronic device of claim 11 , wherein, for the extracting of the color feature maps, the one or more processors are configured to:

perform feature extraction on a first image among the two or more images, based on data generated by feature extraction performed on a second image among the two or more images; and

perform feature extraction on the second image based on the data generated by the feature extraction on the first image.

14 . The electronic device of claim 11 , wherein, for the extracting of the color feature maps, the one or more processors are configured to:

refine a feature map of a layer of a first feature extraction network for a first image using a feature map of a layer of a second feature extraction network for a second image; and

generate a feature map to be propagated to a subsequent layer of the first feature extraction network.

15 . The electronic device of claim 14 , wherein the one or more processors are configured to:

generate an epipolar attention map based on the feature map of the layer of the first feature extraction network and the feature map of the layer of the second feature extraction network; and

generate feature maps for the subsequent layer of the first feature extraction network and a subsequent layer of the second feature extraction network based on the epipolar attention map.

16 . The electronic device of claim 11 , wherein, for the extracting of the color feature maps, the one or more processors are configured to:

perform epipolar attention on three or more images comprising the two or more images by performing feature extraction on a first image among the three or more images based on a result of feature extraction performed on a second image and feature extraction performed on a third image among the three or more images.

17 . The electronic device of claim 11 , wherein, for the extracting of the density feature maps, the one or more processors are configured to:

generate a cost volume based on a correlation for each pyramid level between color feature maps extracted from a first image and color feature maps extracted from a second image; and

generate a density feature map for a corresponding pyramid level based on the cost volume.

18 . The electronic device of claim 11 , wherein, for the generating of the NSR cube information, the one or more processors are configured to:

generate the NSR cube information by storing, in each position in the 3D space, an NSR statistical value of color feature maps and density feature maps extracted from images of a plurality of viewpoints in the same 3D space.

19 . The electronic device of claim 11 , wherein, for the generating of the 2D scene, the one or more processors are configured to:

determine an NSR parameter from the NSR cube information for each of positions in the 3D space along a view direction from a pixel of the 2D scene;

determine a pixel value of the pixel by performing volume rendering based on NSR parameters of the positions in the 3D space along the view direction; and

reconstruct the 2D scene by performing volume rendering on pixels of the 2D scene.

20 . A processor-implemented method, comprising:

generating feature maps for a layer of a first feature extraction network and a layer of a second feature extraction network, respectively based on a first image and a second image;

generating an epipolar attention map based on the feature map of the layer of the first feature extraction network and the feature map of the layer of the second feature extraction network;

generating subsequent feature maps for a subsequent layer of the first feature extraction network and a subsequent layer of the second feature extraction network, based on the epipolar attention map;

determining a three-dimensional (3D) space based on the subsequent feature maps; and

generating a two-dimensional (2D) scene of a field of view (FOV) different from a FOV of the first image and the second image based on the determined 3D space.