Rig-space neural rendering of digital assets

1. A computer-implemented method for generating image data of a scene including a three-dimensional (3D) animatable asset, the method comprising:

accessing a machine learning model that has been trained via first image data of the 3D animatable asset generated by rendering movements of the 3D animatable asset based on first rig vector data that is associated with a plurality of poses of an animation rig usable to deform the 3D animatable asset via a plurality of control points included in the animation rig, wherein at least one of the movements of the 3D animatable asset is rendered from a plurality of virtual camera views;

receiving second rig vector data that includes a plurality of rig parameter values associated with the plurality of control points included in the animation rig; and

generating, via the machine learning model, second image data of the 3D animatable asset based on the second rig vector data, wherein generating the second image data comprises inputting a one-dimensional (1D) array of the plurality of rig parameter values included in the second rig vector data into the machine learning model that outputs the second image data.

2. The computer-implemented method of claim 1 , wherein the second rig vector data includes positional x, y, z data.

3. The computer-implemented method of claim 1 , wherein the second rig vector data includes at least one of position data or orientation data.

4. The computer-implemented method of claim 1 , wherein the second image data includes at least one of albedo image data, surface normal image data, depth image data, or mask image data.

5. The computer-implemented method of claim 1 , wherein the machine learning model includes a plurality of layers, and wherein the machine learning model has been trained by:

training a first layer included in the plurality of layers; and

subsequent to training the first layer, training a second layer included in the plurality of layers that receives an input from the first layer.

6. The computer-implemented method of claim 5 , wherein the second layer includes a convolution block that doubles a spatial resolution of the first layer.

7. The computer-implemented method of claim 1 , wherein a first branch of the machine learning model generates albedo image data, and wherein a second branch of the machine learning model generates surface normal image data, depth image data, and mask image data.

8. The computer-implemented method of claim 7 , wherein at least one of the surface normal image data or the mask image data is derived from the depth image data.

9. The computer-implemented method of claim 1 , further comprising compositing the second image data with image data associated with one or more additional 3D animatable assets to generate a composited scene.

10. The computer-implemented method of claim 9 , wherein the second image data includes surface normal image data, and wherein compositing the second image data with the image data associated with the one or more additional 3D animatable assets comprises applying a light model to the 3D animatable asset based on the surface normal image data.

11. The computer-implemented method of claim 1 , wherein

the second rig vector data includes at least one of a virtual camera position or a virtual camera orientation.

12. A computer-implemented method for generating image data of a scene including a three-dimensional (3D) animatable asset, the method comprising:

accessing a machine learning model that has been trained via first image data of the 3D animatable asset generated by rendering movements of the 3D animatable asset based on first rig vector data that is associated with a plurality of rig poses, wherein at least one of the movements of the 3D animatable asset is rendered from a plurality of virtual camera views;

receiving second rig vector data that includes a plurality of rig parameter values;

generating, via the machine learning model, second image data of the 3D animatable asset based on the second rig vector data, wherein generating the second image data comprises inputting the plurality of rig parameter values included in the second rig vector data into the machine learning model; and

compositing the second image data with image data associated with one or more additional 3D animatable assets to generate a composited scene,

wherein the second image data includes first albedo image data and first depth image data, and

wherein compositing the second image data with the image data associated with

the one or more additional 3D animatable assets comprises:

determining a difference between the first depth image data with second depth image data associated with a second 3D animatable asset included in the one or more additional 3D animatable assets; and

displaying at least one of the first albedo image data or the second albedo image data associated with the second 3D animatable asset based on the difference.

13. One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:

accessing a machine learning model that has been trained via first image data of a 3D animatable asset generated by rendering movements of the 3D animatable asset based on first rig vector data that is associated with a plurality of poses of an animation rig usable to deform the 3D animatable asset via a plurality of control points included in the animation rig, wherein at least one of the movements of the 3D animatable asset is rendered from a plurality of virtual camera views;

receiving second rig vector data that includes a plurality of rig parameter values associated with the plurality of control points included in the animation rig; and

generating, via the machine learning model, second image data of the 3D animatable asset based on the second rig vector data, wherein generating the second image data comprises inputting a one-dimensional (1D) array of the plurality of rig parameter values included in the second rig vector data into the machine learning model that outputs the second image data.

14. The one or more non-transitory computer-readable storage media of claim 13 , wherein the second rig vector data includes positional x, y, z data.

15. The one or more non-transitory computer-readable storage media of claim 13 , wherein the second rig vector data includes at least one of position data or orientation data.

16. The one or more non-transitory computer-readable storage media of claim 13 , wherein the second image data includes at least one of albedo image data, surface normal image data, depth image data, or mask image data.

17. The one or more non-transitory computer-readable storage media of claim 13 , wherein the machine learning model includes a plurality of layers, and wherein the machine learning model has been trained by:

training a first layer included in the plurality of layers; and

subsequent to training the first layer, training a second layer included in the plurality of layers that receives an input from the first layer.

18. The one or more non-transitory computer-readable storage media of claim 17 , wherein the second layer includes a convolution block that doubles a spatial resolution of the first layer.

19. The one or more non-transitory computer-readable storage media of claim 13 , wherein a first branch of the machine learning model generates albedo image data, and wherein a second branch of the machine learning model generates surface normal image data, depth image data, and mask image data.

20. The one or more non-transitory computer-readable storage media of claim 19 , wherein at least one of the surface normal image data or the mask image data is derived from the depth image data.

21. A system, comprising:

a memory that includes instructions; and

a processor that is coupled to the memory and, when executing the instructions:

accesses a machine learning model that has been trained via first image data of a 3D animatable asset generated by rendering movements of the 3D animatable asset based on first rig vector data that is associated with a plurality of poses of an animation rig usable to deform the 3D animatable asset via a plurality of control points included in the animation rig, wherein at least one of the movements of the 3D animatable asset is rendered from a plurality of virtual camera views,

receives second rig vector data that includes a plurality of rig parameter values associated with the plurality of control points included in the animation rig, and

generates, via the machine learning model, second image data of the 3D animatable asset based on the second rig vector data, wherein generating the second image data comprises inputting a one-dimensional (1D) array rcprcscntation of the plurality of rig parameter values included in the second rig vector data into the machine learning model that outputs the second image data.