Radiance field gradient scaling for unbiased near-camera training
Methods and systems disclosed herein relate generally to radiance field gradient scaling for unbiased near-camera training. In a method, a processing device accesses an input image of a three-dimensional environment comprising a plurality of pixels, each pixel comprising a pixel color. The processing device determines a camera location based on the input image and a ray from the camera location in a direction of a pixel. The processing device integrates sampled information from a volumetric representation along the ray from the camera location to obtain an integrated color. The processing device trains a machine learning model configured to predict a density and a color, comprising minimizing a loss function using a scaling factor that is determined based on a distance between the camera location and a point along the ray. The processing device outputs the trained ML model for use in rendering an output image.
1. A method comprising one or more computing devices performing operations comprising:
accessing an input image of a three-dimensional (3D) environment, the input image comprising a plurality of pixels, wherein each pixel of the plurality of pixels comprises a pixel color;
determining a camera location based on the input image of the 3D environment;
determining a ray from the camera location in a direction of a pixel of the plurality of pixels;
integrating sampled information from a volumetric representation of the 3D environment along the ray from the camera location to obtain an integrated color corresponding to the pixel;
training a machine learning (ML) model configured to predict a density and a color of the 3D environment, the training comprising minimizing a loss function using a scaling factor that is determined based on a distance between the camera location and a point along the ray, the loss function defined based on a difference between the integrated color and the pixel color of the pixel; and
outputting the trained ML model for use in rendering an output image of the 3D environment.
2. The method of claim 1 , wherein the scaling factor is determined further based on a scene scale factor, wherein the scene scale factor is based on the distance between the camera location and a feature of the 3D environment.
3. The method of claim 1 , wherein the sampled information from the volumetric representation of the 3D environment along the ray from the camera location is generated by:
determining a plurality of ray locations along the ray; and
for each ray location of the plurality of ray locations along the ray:
determining a sampled density and a sampled color from the volumetric representation associated with the ray location.
4. The method of claim 3 , wherein minimizing the loss function using the scaling factor that is determined based on the distance between the camera location and the point along the ray comprises:
for each ray location of the plurality of ray locations along the ray:
determining a point-wise gradient of the ML model, based on the difference between the integrated color and the pixel color of the pixel; and
scaling the point-wise gradient using the scaling factor based on the distance between the camera location and the point along the ray to obtain a scaled point-wise gradient;
aggregating the scaled point-wise gradients determined for each ray location of the plurality of ray locations to obtain an aggregated scaled gradient; and
applying the aggregated scaled gradient to one or more parameters of the ML model.
5. The method of claim 4 , wherein the scaling factor is the minimum of 1 and a distance term based on the distance between the camera location and the point along the ray.
6. The method of claim 1 , wherein the ML model is a multilayer perceptron (MLP).
7. The method of claim 1 , wherein the volumetric representation comprises one of an MLP, a voxel hash-grid, a tensor decomposition, or voxels.
8. A system, comprising:
a memory component; and
one or more processing devices coupled to the memory component configured to perform operations comprising:
accessing a trained machine learning (ML) model, wherein the trained ML model is trained to predict a density and a color of a 3D environment by minimizing a loss function based on a plurality of scaling factors, wherein each of the plurality of scaling factors are determined based on a distance between a first camera location and a point along a first ray projected from the first camera location towards a pixel in an input image comprising a pixel color;
receiving a second camera location and a camera direction;
using the trained ML model, determining a plurality of densities and colors of the 3D environment from a perspective of the second camera location at a plurality of respective points sampled along a second ray projected from the second camera location in the direction of the camera direction; and
aggregating the plurality of densities and colors of the 3D environment to generate an output pixel comprising an integrated color that represents the 3D environment.
9. The system of claim 8 , wherein each scaling factor is determined further based on a scene scale factor, wherein the scene scale factor is based on a distance scale that characterizes the 3D environment.
10. The system of claim 8 , wherein predicting the density and the color of a 3D environment comprises:
determining a plurality of ray locations along the first ray; and
for each ray location of the plurality of ray locations along the first ray:
determining a sampled density and a sampled color from a volumetric representation associated with the ray location.
11. The system of claim 10 , wherein minimizing the loss function based on the plurality of scaling factors, wherein each of the plurality of scaling factors are determined based on the distance between the first camera location and the point along the first ray projected from the first camera location towards the pixel in the input image comprising the pixel color comprises:
for each ray location of the plurality of ray locations along the first ray:
determining a point-wise gradient of the ML model, based on a difference between the integrated color and the pixel color of the pixel; and
scaling the point-wise gradient using the scaling factor based on the distance between the first camera location and the point along the first ray to obtain a scaled point-wise gradient;
aggregating the scaled point-wise gradients determined for each ray location of the plurality of ray locations to obtain an aggregated scaled gradient; and
applying the aggregated scaled gradient to one or more parameters of the ML model.
12. The system of claim 8 , wherein the ML model is a fully-connected neural network.
13. The system of claim 8 , wherein determining the plurality of densities and colors of the 3D environment from the perspective of the second camera location at the plurality of respective points sampled along the second ray projected from the second camera location is based on a volumetric representation, wherein the volumetric representation comprises one of a multilayer perceptron (MLP), a voxel hash-grid, a tensor decomposition, or voxels.
14. A computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more processing devices to perform actions including:
accessing a machine learning (ML) model trained via a training process, wherein the ML model is configured to predict a color of a 3D environment, wherein the training process comprises:
a step for determining a plurality of densities and colors of the 3D environment from a training perspective of a first camera location at a plurality of respective points sampled along a first ray projected from the first camera location;
a step for aggregating the plurality of densities and colors of the 3D environment to generate an integrated pixel color; and
a step for minimizing a loss function of the ML model using a scaling factor that is determined based on a distance between the first camera location and a point along the first ray projected from the first camera location towards the 3D environment; and
receiving a second camera location and a camera direction;
using the trained ML model, determining a plurality of densities and colors of the 3D environment from a perspective of the second camera location at a plurality of respective points sampled along a second ray projected from the second camera location in the direction of the camera direction; and
aggregating the plurality of densities and colors of the 3D environment to generate an output pixel comprising an integrated color that represents the 3D environment.
15. The computer program product of claim 14 , wherein the scaling factor is determined further based on a scene scale factor, wherein the scene scale factor is based on the distance between the first camera location and a feature of the 3D environment.
16. The computer program product of claim 14 , wherein the step for determining the plurality of densities and colors of the 3D environment from the training perspective of the first camera location at the plurality of respective points sampled along the first ray projected from the first camera location comprises:
determining a plurality of ray locations along the first ray; and
for each ray location of the plurality of ray locations along the first ray:
determining a sampled density and a sampled color based on a volumetric representation associated with the ray location.
17. The computer program product of claim 16 , wherein minimizing the loss function using the scaling factor that is determined based on the distance between the first camera location and the point along the first ray comprises:
for each ray location of the plurality of ray locations along the first ray:
determining a point-wise gradient of the ML model, based on a difference between the integrated color and the pixel color of the pixel; and
scaling the point-wise gradient using the scaling factor based on the distance between the first camera location and the point along the first ray to obtain a scaled point-wise gradient;
aggregating the scaled point-wise gradients determined for each ray location of the plurality of ray locations to obtain an aggregated scaled gradient; and
applying the aggregated scaled gradient to one or more parameters of the ML model.
18. The computer program product of claim 17 , wherein the scaling factor is the minimum of 1 and the square of the distance between the first camera location and the point along the first ray.
19. The computer program product of claim 14 , wherein the ML model is a multilayer perceptron (MLP).
20. The computer program product of claim 14 , wherein determining the plurality of densities and colors of the 3D environment from the perspective of the second camera location at the plurality of respective points sampled along the second ray projected from the second camera location is based on a volumetric representation, wherein the volumetric representation comprises one of an MLP, a voxel hash-grid, a tensor decomposition, or voxels.