Methods and systems for training quantized neural radiance field
A computer-implemented method includes encoding a radiance field of an object onto a machine learning model; conducting, based on a set of training images of the object, a training process on the machine learning model to obtain a trained machine learning model, wherein the training process includes a first training process using a plurality of first test sample points followed by a second training process using a plurality of second test sample points located within a threshold distance from a surface region of the object; obtaining target view parameters indicating a view direction of the object; obtaining a plurality of rays associated with a target image of the object; obtaining render sample points on the plurality of rays associated with the target image; and rendering, by inputting the render sample points to the trained machine learning model, colors associated with the pixels of the target image.
1 . A computer-implemented method, comprising:
encoding, by a computer device, a radiance field of an object onto a machine learning model;
conducting, by the computer device and based on a set of training images of the object, a training process on the machine learning model to obtain a trained machine learning model, wherein the training process comprises a first training process using a plurality of first test sample points followed by a second training process using a plurality of second test sample points, each of the plurality of first test sample points and the plurality of second test sample points located on a plurality of training rays derived from the training images, the plurality of second test sample points located within a threshold distance from a surface region of the object;
obtaining target view parameters indicating a viewpoint and a view direction of the object;
obtaining, based on the target view parameters, a plurality of sample rays associated with a target image of the object, the target image associated with the viewpoint and the view direction;
obtaining, by the computer device, render sample points on the plurality of sample rays; and
rendering, by inputting the render sample points to the trained machine learning model, colors associated with pixels of the target image.
2 . The computer-implemented method according to claim 1 , wherein the radiance field comprises a three-dimensional (3D) rendering space enclosing the object.
3 . The computer-implemented method according to claim 2 , wherein the machine learning model is a fully connected neural network comprising at least one node, each having an associated weight.
4 . The computer-implemented method according to claim 3 , wherein the machine learning model is configured to accept a position vector and a direction vector of a point in the 3D rendering space as an input and output a density and a radiance of the point, the position vector indicating a location of the point with respect to the viewpoint, and the direction vector indicating a relative direction of the point with respect to the viewpoint.
5 . The computer-implemented method according to claim 4 , wherein during the second training process, values of the weights, elements of the position vectors, elements of the direction vectors, the densities, and the radiances are quantized.
6 . The computer-implemented method according to claim 5 , wherein the step of obtaining the render sample points on the plurality of sample rays comprises: obtaining the render sample points, wherein the render sample points are located within the threshold distance from the surface region of the object.
7 . The computer-implemented method according to claim 5 , wherein the surface region of the object is obtained using a marching cube technique based on the set of training images, and the surface region is in a form of triangular mesh.
8 . The computer-implemented method according to claim 5 , wherein the threshold distance is 2 cm.
9 . A device comprising:
a processor; and
a memory storing instructions executable by the processor, wherein the instructions are executed by the processor, the instructions cause the processor to perform operations, comprising:
encoding a radiance field of an object onto a machine learning model:
conducting, based on a set of training images of the object, a training process on the machine learning model to obtain a trained machine learning model, wherein the training process comprises a first training process using a plurality of first test sample points followed by a second training process using a plurality of second test sample points, each of the plurality of first test sample points and the plurality of second test sample points located on a plurality of training rays derived from the training images, the plurality of second test sample points located within a threshold distance from a surface region of the object;
obtaining target view parameters indicating a viewpoint and a view direction of the object;
obtaining, based on the target view parameters, a plurality of sample rays associated with a target image of the object, the target image associated with the viewpoint and the view direction;
obtaining render sample points on the plurality of sample rays; and
rendering, by inputting the render sample points to the trained machine learning model, colors associated with pixels of the target image.
10 . The device according to claim 9 , wherein the radiance field comprises a three-dimensional (3D) rendering space enclosing the object.
11 . The device according to claim 10 , wherein the machine learning model is a fully connected neural network comprising at least one node, each having an associated weight.
12 . The device according to claim 11 , wherein the machine learning model is configured to accept a position vector and a direction vector of a point in the 3D rendering space as an input and output a density and a radiance at the point, the position vector indicating a location of the point with respect to the viewpoint, and the direction vector indicating a relative direction of the point with respect to the viewpoint.
13 . The device according to claim 12 , wherein during the second training process, values of the weights, elements of the position vectors, elements of the direction vectors, the densities, and the radiances are quantized.
14 . The device according to claim 13 , wherein the step of obtaining the render sample points on the plurality of sample rays comprises: obtaining the render sample points, wherein the render sample points are located within the threshold distance from the surface region of the object.
15 . The device according to claim 13 , wherein the surface region of the object is obtained using a marching cube technique based on the set of training images, and the surface region is in a form of triangular mesh.
16 . The device according to claim 13 , wherein the threshold distance is 2 cm.
17 . A non-transitory storage medium of storing instructions executable by a process, wherein, the instructions are executed by the processors, the instructions cause the processor to perform operations, comprising:
encoding a radiance field of an object onto a machine learning model;
conducting, based on a set of training images of the object, a training process on the machine learning model to obtain a trained machine learning model, wherein the training process comprises a first training process using a plurality of first test sample points followed by a second training process using a plurality of second test sample points, each of the plurality of first test sample points and the plurality of second test sample points located on a plurality of training rays derived from the training images, the plurality of second test sample points located within a threshold distance from a surface region of the object;
obtaining target view parameters indicating a viewpoint and a view direction of the object;
obtaining, based on the target view parameters, a plurality of sample rays associated with a target image of the object, the target image associated with the viewpoint and the view direction;
obtaining render sample points on the plurality of sample rays associated with the target image; and
rendering, by inputting the render sample points to the trained machine learning model, colors associated with pixels of the target image.
18 . The non-transitory storage medium according to claim 17 , wherein the radiance field comprises a three-dimensional (3D) rendering space enclosing the object.
19 . The non-transitory storage medium according to claim 18 , wherein the machine learning model is a fully connected neural network comprising at least one node, each having an associated weight.
20 . The non-transitory storage medium according to claim 19 , wherein the machine learning model is configured to accept a position vector and a direction vector of a point in the 3D rendering space as an input and output a density and a radiance of the point, the position vector indicating a location of the point with respect to the viewpoint, and the direction vector indicating a relative direction of the point with respect to the viewpoint.