Computing images of dynamic scenes
Computing an output image of a dynamic scene. A value of E is selected which is a parameter describing desired dynamic content of the scene in the output image. Using selected intrinsic camera parameters and a selected viewpoint, for individual pixels of the output image to be generated, the method computes a ray that goes from a virtual camera through the pixel into the dynamic scene. For individual ones of the rays, sample at least one point along the ray. For individual ones of the sampled points, a viewing direction being a direction of the corresponding ray, and E, query a machine learning model to produce colour and opacity values at the sampled point with the dynamic content of the scene as specified by E. For individual ones of the rays, apply a volume rendering method to the colour and opacity values computed along that ray, to produce a pixel value of the output image.
1. A computer-implemented method of training a machine learning model, the method comprising:
accessing a plurality of training images of a dynamic scene, the training images having been captured from a plurality of different viewpoints and at a plurality of different times;
for an individual image of the training images:
specifying a viewing direction of a pixel of the individual image according to a viewpoint of a capture device which captured the individual image;
specifying a value of E which is a parameter describing desired dynamic content of the dynamic scene in an output image of the dynamic scene; and
training the machine learning model using supervised learning given the training images such that the machine learning model produces a radiance value of an output three-dimensional point in the dynamic scene, the radiance value of the output three-dimensional point comprising a color value and an opacity value, given points in the dynamic scene, the viewing direction, and the value of E; the training including:
training the machine learning model to generate the color value based on both a location of the output three-dimensional point on a ray and a direction of the ray, and
training the machine learning model to generate the opacity value based on the location of the output three-dimensional point on the ray.
2. The computer-implemented method of claim 1 , further comprising:
the value of E having a type and a format;
the type of the value of E depending on the training images; and
the format of the value of E depending on the training images.
3. The computer-implemented method of claim 1 , wherein the dynamic scene is three-dimensional and comprises a moving object.
4. The computer-implemented method of claim 1 , further comprising, for individual ones of the training images, specifying intrinsic parameter values of the capture device associated with the output image, and a viewpoint for the capture device.
5. The computer-implemented method of claim 1 , wherein the value of E is specified using one or more of: a time when the individual image was captured, a value of parameters of a 3D model of an object in the dynamic scene at the time when the individual image was captured.
6. The computer-implemented method of claim 1 , wherein the machine learning model is a neural network with a plurality of layers, each layer comprising a plurality of nodes where each node has a weight.
7. The computer-implemented method of claim 6 , further comprising modifying the weight using the value of E.
8. An apparatus comprising:
a processor; and
a memory storing instructions that, when executed by the processor, perform a method of training a machine learning model comprising:
accessing a plurality of training images of a dynamic scene, the training images having been captured from a plurality of different viewpoints and at a plurality of different times;
for an individual image of the training images:
specifying a viewing direction of a pixel of the individual image according to a viewpoint of a capture device which captured the individual image;
specifying a value of E which is a parameter describing desired dynamic content of the dynamic scene in an output image of the dynamic scene; and
training the machine learning model using supervised learning given the training images such that the machine learning model produces a radiance value of an output three-dimensional point in the dynamic scene, the radiance value of the output three-dimensional point comprising a color value and an opacity value, given points in the dynamic scene, the viewing direction, and the value of E; the training including:
training the machine learning model to generate the color value based on both a location of the output three-dimensional point on a ray and a direction of the ray, and
training the machine learning model to generate the opacity value based on the location of the output three-dimensional point on the ray.
9. The apparatus of claim 8 , further comprising:
the value of E having a type and a format;
the type of the value of E depending on the training images; and
the format of the value of E depending on the training images.
10. The apparatus of claim 8 , wherein the dynamic scene is three-dimensional and comprises a moving object.
11. The apparatus of claim 8 , further comprising, for individual ones of the training images, specifying intrinsic parameter values of the capture device associated with the output image, and a viewpoint for the capture device.
12. The apparatus of claim 8 , wherein the value of E is specified using one or more of: a time when the individual image was captured, a value of parameters of a 3D model of an object in the dynamic scene at the time when the individual image was captured.
13. The apparatus of claim 8 , wherein the machine learning model is a neural network with a plurality of layers, each layer comprising a plurality of nodes where each node has a weight.
14. The apparatus of claim 13 , further comprising modifying the weight using the value of E.
15. A computer storage medium storing computer executable instructions that upon execution by a processor perform a method of training a machine learning model comprising:
accessing a plurality of training images of a dynamic scene, the training images having been captured from a plurality of different viewpoints and at a plurality of different times;
for an individual image of the training images:
specifying a viewing direction of a pixel of the individual image according to a viewpoint of a capture device which captured the individual image;
specifying a value of E which is a parameter describing desired dynamic content of the dynamic scene in an output image of the dynamic scene; and
training the machine learning model using supervised learning given the training images such that the machine learning model produces a radiance value of an output three-dimensional point in the dynamic scene, the radiance value of the output three-dimensional point comprising a color value and an opacity value, given points in the dynamic scene, the viewing direction, and the value of E; the training including:
training the machine learning model to generate the color value based on both a location of the output three-dimensional point on a ray and a direction of the ray, and
training the machine learning model to generate the opacity value based on the location of the output three-dimensional point on the ray.
16. The computer storage medium of claim 15 , further comprising:
the value of E having a type and a format;
the type of the value of E depending on the training images; and
the format of the value of E depending on the training images.
17. The computer storage medium of claim 15 , wherein the dynamic scene is three-dimensional and comprises a moving object.
18. The computer storage medium of claim 15 , further comprising, for individual ones of the training images, specifying intrinsic parameter values of the capture device associated with the output image, and a viewpoint for the capture device.
19. The computer storage medium of claim 15 , wherein the value of E is specified using one or more of: a time when the individual image was captured, a value of parameters of a 3D model of an object in the dynamic scene at the time when the individual image was captured.
20. The computer storage medium of claim 15 , wherein the machine learning model is a neural network with a plurality of layers, each layer comprising a plurality of nodes where each node has a weight, wherein the weight is modified using the value of E.