Dynamic 3D scene generation
A cage of primitive 3D elements and associated animation data is received. Compute a ray from a virtual camera through a pixel into the cage animated according to the animation data and compute a plurality of samples on the ray. Compute a transformation of the samples into a canonical cage. For each transformed sample, query a plurality of learnt radiance field parameterizations, each learnt on a different deformed state of the 3D scene to obtain color values for each learnt radiance field. For each transformed sample, query a learnt radiance field parameterization of the 3D scene to obtain an opacity value. Compute, for each transformed sample, a weighted combination of the color values, wherein the weights are related to the local features. A volume rendering method is applied to the weighted combinations of the color and the opacity values producing a pixel value.
1. A computer-implemented method of computing an image of a dynamic 3D scene comprising a 3D object, the method comprising:
receiving a description of a deformation of the 3D object, the description comprising a cage of primitive 3D elements and associated animation data from a physics engine or an articulated object model;
for a pixel of the image, computing a ray from a virtual camera through the pixel into the cage animated according to the animation data and computing a plurality of samples on the ray, each sample being a 3D position and view direction in one of the 3D elements;
computing a transformation of the samples into a canonical version of the cage to produce transformed samples and local features describing a volume change between canonical and non-canonical states of the cage;
for each transformed sample, querying a plurality of learnt radiance field parameterizations of the 3D scene, each learnt on a different deformed state of the scene, to obtain a color value from each learnt radiance field;
for each transformed sample, querying a learnt radiance field parameterization of the 3D scene to obtain an opacity value;
computing, for each transformed sample, a weighted combination of the color values, wherein the weights are related to the local features; and
applying a volume rendering method to the weighted combinations of the color values and the opacity values to produce a pixel value of the image.
2. The method of claim 1 wherein the cage of primitive 3D elements represents the 3D object and a volume extending from the 3D object.
3. The method of claim 2 wherein the cage comprises a volumetric mesh with a plurality of volumetric blendshapes and a skeleton.
4. The method of claim 1 further comprising, for one of the transformed samples, rotating a view direction of a ray of the sample prior to querying the plurality of learnt radiance field parameterizations.
5. The method of claim 4 comprising computing a rotation R of the view direction for a small fraction of the primitive 3D elements and propagating the value of R to remaining primitive 3D elements via nearest neighbor interpolation.
6. The method of claim 1 wherein the canonical version of the cage is the cage with specified parameter values of an articulated object model or specified parameters of a physics engine.
7. The method of claim 1 wherein the canonical version of the cage represents a face with a neutral expression including a closed mouth.
8. The method of claim 1 wherein the weights are related to the local features by a softmax-gated local similarity measure that is equivalent to a vector field of the weights discretized on vertices of the cage, wherein the local similarity measure defines the difference between a local measure of volume between canonical and non-canonical states of the cage.
9. The method of claim 1 wherein the learnt radiance field parameterizations are each a cache of associations between 3D points in the canonical version of the cage and color and opacity values, obtained by querying a machine learning model trained using training data comprising images of the dynamic scene from a plurality of viewpoints.
10. The method of claim 9 wherein the images of the dynamic scene from a plurality of viewpoints are obtained at the same time instance or two time instants.
11. The method of claim 1 wherein querying a learnt radiance field parameterization of the 3D scene to obtain an opacity value comprises querying a plurality of learnt radiance field parameterizations of the 3D scene, each learnt on a different deformed state of the scene, to obtain an opacity value from each learnt radiance field, and computing, for each transformed sample, a weighted combination of the opacity values, wherein the weights are related to the local features.
12. The method of claim 1 comprising, during the process of, for each transformed sample, querying each learnt radiance field parameterization of the plurality of learnt radiance field parameterizations of the 3D scene to obtain a color value and an opacity value, using only one radiance field network and increasing a number of sampling bins.
13. The method of claim 1 wherein the primitive 3D elements are tetrahedra, and wherein the method further comprises computing the transformation P of a sample by setting P equal to a normalized distance between a previous and a next intersection of a tetrahedron on a ray, times a sum, at the previous intersection, over four vertices of a tetrahedron of a barycentric coordinate of a vertex times a canonical coordinate of the vertex, plus one minus the normalized distance, times a sum, at the next intersection, over four vertices of the tetrahedron of the barycentric coordinate of a vertex times the canonical coordinate of the vertex.
14. An apparatus comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, perform a method for computing an image of a dynamic 3D scene comprising a 3D object, comprising:
receiving a description of a deformation of the 3D object, the description comprising a cage of primitive 3D elements and associated animation data from a physics engine or an articulated object model;
for a pixel of the image, computing a ray from a virtual camera through the pixel into the cage animated according to the animation data and computing a plurality of samples on the ray, each sample being a 3D position and view direction in one of the 3D elements;
computing a transformation of the samples into a canonical version of the cage to produce transformed samples and local features describing a volume change between canonical and non-canonical states of the cage;
for each transformed sample, querying a plurality of learnt radiance field parameterizations of the 3D scene, each learnt on a different deformed state of the scene, to obtain a color value from each learnt radiance field;
for each transformed sample, querying a learnt radiance field parameterization of the 3D scene to obtain an opacity value;
computing, for each transformed sample, a weighted combination of the color values, wherein the weights are related to the local features; and
applying a volume rendering method to the weighted combinations of the color values and the opacity values to produce a pixel value of the image.
15. A computer-implemented method of computing an image of a dynamic 3D scene comprising a 3D object, the method comprising:
receive a description of a deformation of the 3D object, the description comprising 3D elements;
for a pixel of the image, computing a ray from a virtual camera through the pixel into the description and computing a plurality of samples on the ray, each sample being a 3D position and view direction in one of the 3D elements;
compute a transformation of the samples into a canonical space to produce transformed samples and local features;
for each transformed sample, query a plurality of caches of associations between 3D points in the canonical space and color and opacity values to obtain color values, each cache produced using images of a different deformed state of the scene;
for each transformed sample, querying a cache of associations between 3D points in the canonical space and color and opacity values to obtain an opacity value;
compute, for each transformed sample, a weighted combination of the color values, wherein the weights are related to the local features; and
apply a volume rendering method to the weighted combinations of the color values and the opacity values to produce a pixel value of the image.
16. The method of claim 15 further comprising one or more of: storing the image, transmitting the image to a computer game application, transmitting the image to a telepresence application, inserting the image into a virtual webcam stream, transmitting the image to a head mounted display.
17. The method of claim 16 comprising using an object tracker to detect parameter values of a model of a 3D object depicted in a video and using the detected parameter values and the model to compute the description of the deformation of the 3D object.
18. The method of claim 16 comprising using a physics engine to specify the description.
19. The method of claim 16 wherein the 3D elements are any of: tetrahedra, spheres, cuboids.
20. The method of claim 16 wherein querying a learnt radiance field parameterization of the 3D scene to obtain an opacity value comprises querying a plurality of learnt radiance field parameterizations of the 3D scene, each learnt on a different deformed state of the scene, to obtain an opacity value from each learnt radiance field, and computing, for each transformed sample, a weighted combination of the opacity values, wherein the weights are related to the local features.