Method and device for representing rendered scenes
Disclosed are a method and device for representing rendered scenes. A data processing method of training a neural network model includes obtaining spatial information of sampling data, obtaining one or more volume-rendering parameters by inputting the spatial information of the sampling data to the neural network model, obtaining a regularization term based on a distribution of the volume-rendering parameters, performing volume rendering based on the volume-rendering parameters, and training the neural network model to minimize a loss function determined based on the regularization term and based on a difference between a ground truth image and an image that is estimated according to the volume rendering.
1 . A method of training a neural network model for scene representation, the method comprising:
obtaining spatial information of sampling data, the spatial information including information about points sampled from a ray or from a three-dimensional model;
obtaining volume-rendering parameters by inputting the spatial information of the sampling data to the neural network model, which generates the volume-rendering parameters, the volume-rendering parameters corresponding to the points, respectively;
obtaining a regularization term quantifying a statistical distribution of the volume-rendering parameters;
performing volume rendering based on the volume-rendering parameters; and
training the neural network model to minimize a loss function, wherein the loss function is determined based on the regularization term and is determined based on a difference between a ground truth image and an image that is estimated according to the volume rendering.
2 . The method of claim 1 , wherein the training the neural network model comprises:
training the neural network model such that the distribution of the volume-rendering parameters has a predetermined feature.
3 . The method of claim 2 , wherein the training the neural network model comprises:
training the neural network model such that the distribution of the volume-rendering parameters is clustered on a surface of a scene.
4 . The method of claim 1 , wherein the obtaining the regularization term comprises:
obtaining the regularization term based on a metric quantifying a feature of the distribution of the volume-rendering parameters.
5 . The method of claim 1 , wherein the obtaining the regularization term comprises:
obtaining an entropy measure corresponding to the distribution of the volume-rendering parameters; and
obtaining an information potential corresponding to the volume-rendering parameters, based on the entropy measure.
6 . The method of claim 5 , wherein the training the neural network model is performed such that the information potential is maximized.
7 . The method of claim 1 , wherein
the loss function is determined by adding a second loss function to a first loss function, wherein
the first loss function is determined based on the difference between the ground truth image and the image is estimated through the volume rendering and the second loss function is determined based on the regularization term.
8 . The method of claim 1 , wherein the obtaining the regularization term comprises:
obtaining the distribution of the volume-rendering parameters corresponding to sample point of sampling data included in a set of sample points in a predetermined area;
obtaining a statistical value of the distribution of the volume-rendering parameters corresponding to the sample points; and
determining the statistical value to be the regularization term.
9 . The method of claim 1 , wherein the obtaining the spatial information of the sampling data comprises obtaining spatial information of a ray and obtaining sampling information.
10 . A scene representation method comprising:
obtaining spatial information of sampling data, the sampling data sampled from a three-dimensional (3D) model, the spatial information including information about points sampled from a ray or from a three-dimensional model; and
performing volume rendering of the 3D model by inputting the spatial information of the sampling data to a neural network model that generates volume rendering parameters, wherein
the spatial information of the sampling data is determined based on a quantification of an amount of information in the volume rendering parameters.
11 . The method of claim 10 , wherein
the neural network model is trained to transform the distribution of the volume rendering parameters to perform the volume rendering.
12 . The method of claim 10 , wherein
the spatial information of sampling data comprises at either position information or information on a number of sample points in the sampling data.
13 . The method of claim 10 , wherein
the obtaining the spatial information of sampling data further comprises obtaining a depth map corresponding to a scene of the 3D model.
14 . The method of claim 10 , wherein
the obtaining the spatial information of sampling data further comprises obtaining information on a surface of the 3D model.
15 . The method of claim 10 , wherein
the obtaining the spatial information of the sampling data comprises obtaining spatial information of a ray and obtaining sampling information.
16 . A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .
17 . An electronic device comprising:
one or more processors;
memory storing instructions configured to, when executed by the one or more processors, cause the one or more processors to:
obtain spatial information of sampling data, the spatial information including information about points sampled from a ray or from a three-dimensional model,
obtain volume-rendering parameters by inputting the spatial information of the sampling data to a neural network model, which generates the volume-rendering parameters, the volume-rendering parameters corresponding to the points, respectively,
obtain a regularization term quantifying an amount of information in the volume-rendering parameters,
perform volume rendering based on the volume-rendering parameters, and
train the neural network model to minimize a loss function, wherein the loss function is determined based on the regularization term and is determined based on a difference between a ground truth image and an image that is estimated according to the volume rendering.
18 . The electronic device of claim 17 , wherein the instructions are further configured to cause the one or more processors to train the neural network model such that the distribution of the volume-rendering parameters has a predetermined feature.
19 . The electronic device of claim 17 , wherein the instructions are further configured to cause the one or more processors to train the neural network model such that the distribution of the volume-rendering parameters is clustered on a surface of a scene.
20 . The electronic device of claim 17 , wherein the instructions are further configured to cause the one or more processors to obtain the regularization term based on a metric for quantifying a feature of the distribution of the volume-rendering parameters.
21 . An electronic device comprising:
one or more processors configured to
obtain spatial information of sampling data, the sampling data sampled from a ray or a three-dimensional (3D) model, and
perform volume rendering of the 3D model by inputting the spatial information of the sampling data to a neural network model that generates volume rendering parameters, wherein
the spatial information of the sampling data is determined based on a regularization term quantifying an amount of information in the volume rendering parameters.