Virtual viewpoint image synthesis device and method using quasi-uniform spherical coordinate grid

1 . An image synthesis device capable of synthesizing virtual viewpoint images, the image synthesis device comprising:

a memory storing a virtual viewpoint image synthesis program; and

a processor configured to execute the virtual viewpoint image synthesis program,

wherein the virtual viewpoint image synthesis program receives images of multiple viewpoints obtained by filming around a user, position information on the user, and information on a direction in which a scene is viewed, estimates a color and a density by inputting the position information of the user and the direction in which an object is viewed to a neural radiance field model constructed by using the images of the multiple viewpoints, and synthesizes images of a virtual viewpoint by performing volume rendering by using the estimated color and the estimated density,

wherein the neural radiance field model is constructed based on a spherical coordinate system,

wherein the spherical coordinate system includes a sum of a Yin grid and a Yang grid and has a grid structure that increases exponentially in a radial direction, and

wherein, in the spherical coordinate system, the Yin grid is defined by an equation below, and the Yang grid is identical to the Yin grid and has an axis located at an equator of the Yin grid through rotation transformation (π/4≤θ≤3π/4)∩(−3π/4≤φ≤3π/4) where Θ indicates colatitude and φ indicates longitude.

2 . The image synthesis device of claim 1 , wherein

the neural radiance field model estimates the density through hierarchical sampling, obtains a coarse grid from the spherical coordinate system, performs coarse sampling based on the coarse grid to obtain a first number of density values, and then obtains a second number of density values through precise sampling.

3 . The image synthesis device of claim 1 , wherein

the neural radiance field model performs the volume rendering by considering an environment lighting element when performing the volume rendering.

4 . The image synthesis device of claim 1 ,

wherein the neural radiance field model represents the scene using an explicit grid structure (G σ ) that stores density values and an explicit grid structure (G a ) that stores latent vectors indicative of the color, and wherein the explicit grid structure (G σ ) and the explicit grid structure (G a ) are decomposed into vector-matrix components corresponding to axes of the spherical coordinate system.

5 . An image synthesis method of virtual viewpoint image synthesis device, the image synthesis method comprising:

receiving, by the virtual viewpoint image synthesis device, images of multiple viewpoints obtained by filming around a user, position information on the user, and information on a direction in which a scene is viewed;

estimating, by the virtual viewpoint image synthesis device, a color and a density by inputting the position information of the user and the direction in which an object is viewed to a neural radiance field model constructed by using the images of the multiple viewpoints; and

synthesizing, by the virtual viewpoint image synthesis device, images of a virtual viewpoint by performing volume rendering by using the estimated color and the estimated density,

wherein the neural radiance field model is constructed based on a spherical coordinate system, and

wherein the spherical coordinate system includes a sum of a Yin grid and a Yang grid and has a grid structure that increases exponentially in a radial direction, and

wherein in the spherical coordinate system, the Yin grid is defined by an equation below, and the Yang grid is identical to the Yin grid and has an axis located at an equator of the Yin grid through rotation transformation (π/4≤θ≤3π/4)∩(−3π/4≤φ≤3π/4) where Θ indicates colatitude and φ indicates longitude.

6 . The image synthesis method of claim 5 , wherein,

in the estimating of the color and the density, the density is estimated through hierarchical sampling, a coarse grid is obtained from the spherical coordinate system, coarse sampling is performed based on the coarse grid to obtain a first number of density values, and then a second number of density values are obtained through precise sampling.

7 . The image synthesis method of claim 5 , wherein,

in the synthesizing of the images of the virtual viewpoint by performing the volume rendering, the volume rendering is performed by considering an environment lighting element when performing the volume rendering.

8 . The image synthesis method of claim 5 ,

wherein the neural radiance field model represents the scene using an explicit grid structure (G σ ) that stores density values and an explicit grid structure (G a ) that stores latent vectors indicative of the color, and

wherein the explicit grid structure (G σ ) and the explicit grid structure (G a ) are decomposed into vector-matrix components corresponding to axes of the spherical coordinate system.

9 . A non-transitory computer-readable recording medium on which a computer program for performing the image synthesis method according to claim 5 is recorded.