Avatar generation based on driving views

1. A computing system, comprising:

a processor and memory of a computing device, the processor being configured to execute an avatar generation program using portions of memory to:

obtain one or more driving views;

calculate a skeletal pose of a user;

generate a coarse human mesh representing the user based on a template mesh and the skeletal pose of the user;

construct a UV positional map based on the coarse human mesh;

construct a texture map based on the one or more driving views and the coarse human mesh;

extract a plurality of image features from the texture map, the image features being aligned to the UV positional map;

extract a plurality of pose features from the UV positional map, the pose features being aligned to the UV positional map;

generate a plurality of pose-image features based on the UV map-aligned image features and UV map-aligned pose features; and

render an avatar based on the plurality of pose-image features.

2. The computing system of claim 1 , wherein

the one or more driving views are transformed into a partial texture map comprising a plurality of unwrapped texture maps in UV space; and

the plurality of unwrapped texture maps are unioned and averaged to construct the texture map based on the one or more driving views and the coarse human mesh.

3. The computing system of claim 1 , wherein

the plurality of pose features are extracted from the UV positional map via a pose convolutional neural network which regresses the UV positional map into the pose features aligned to the UV positional map; and

the plurality of image features are extracted from the texture map via an image convolutional neural network which regresses the texture map into the image features aligned to the UV positional map.

4. The computing system of claim 1 , wherein each of the one or more driving views is a pixel image with color channels and no depth channels.

5. The computing system of claim 1 , wherein

the UV positional map is constructed by projecting each surface point on the coarse human mesh from 3D space to a UV surface manifold; and

each pixel on the UV positional map describes a relative location of a point on the UV surface manifold.

6. The computing system of claim 1 , wherein the coarse human mesh is generated using a Linear Blend Skinning (LBS) function on the template mesh of the user and the skeletal pose of the user.

7. The computing system of claim 1 , wherein the plurality of pose-image features are generated via a neural texture model which receives input of the UV map-aligned image features and the UV map-aligned pose features and outputs concatenated pose-image feature vectors.

8. The computing system of claim 7 , wherein the pose-image features include texture information indicating where wrinkles form in clothing of the user.

9. The computing system of claim 1 , wherein the processor is further configured to:

generate textural features and multi-channel volumetric features from the pose-image features, channels of the multi-channel volumetric features corresponding to color channels of one or a plurality of pixel images of the one or more driving views; and

fuse the textural features and the multi-channel volumetric features to render the avatar.

10. The computing system of claim 1 , wherein the processor is further configured to:

obtain a target surface normal of the coarse human mesh and project the target surface normal into UV space, wherein

the surface normal of the coarse human mesh is supervised with an L1 loss.

11. A computerized method, comprising:

obtaining one or more driving views;

calculating a skeletal pose of a user;

generating a coarse human mesh representing the user based on a template mesh and the skeletal pose of the user;

constructing a UV positional map based on the coarse human mesh;

constructing a texture map based on the one or more driving views and the coarse human mesh;

extracting a plurality of image features from the texture map, the image features being aligned to the UV positional map;

extracting a plurality of pose features from the UV positional map, the pose features being aligned to the UV positional map;

generating a plurality of pose-image features based on the UV map-aligned image features and the UV map-aligned pose features; and

rendering an avatar based on the plurality of pose-image features.

12. The method of claim 11 , wherein

the one or more driving views are transformed into a partial texture map comprising a plurality of unwrapped texture maps in UV space; and

the plurality of unwrapped texture maps are unioned and averaged to construct the texture map based on the one or more driving views and the coarse human mesh.

13. The method of claim 11 , wherein

the plurality of pose features are extracted from the UV positional map via a pose convolutional neural network which regresses the UV positional map into the pose features aligned to the UV positional map; and

the plurality of image features are extracted from the texture map via an image convolutional neural network which regresses the texture map into the image features aligned to the UV positional map.

14. The method of claim 11 , wherein the one or more driving views is one or a plurality of pixel images with color channels and no depth channels.

15. The method of claim 11 , wherein

the UV positional map is constructed by projecting each surface point on the coarse human mesh from 3D space to a UV surface manifold; and

each pixel on the UV positional map describes a relative location of a point on the UV surface manifold.

16. The method of claim 11 , wherein the coarse human mesh is generated using a Linear Blend Skinning (LBS) function on the template mesh of the user and the skeletal pose of the user.

17. The method of claim 11 , wherein the plurality of pose-image features are generated via a neural texture model which receives input the UV map-aligned image features and the UV map-aligned pose features and outputs concatenated pose-image feature vectors.

18. The method of claim 17 , wherein the pose-image features include texture information indicating where wrinkles form in clothing of the user.

19. The method of claim 11 , further comprising:

generating textural features and multi-channel volumetric features from the pose-image features, channels of the multi-channel volumetric features corresponding to color channels of one or a plurality of pixel images of the one or more driving views; and

fusing the textural features and the multi-channel volumetric features to render the avatar and a mask corresponding to the avatar.

20. A computing system, comprising:

a camera, a processor, and memory of a computing device, the processor being configured to execute an avatar generation program using portions of memory to:

obtain one or more driving views of a user via the camera, the one or more driving views being one or a plurality of pixel images with color channels and no depth channels;

calculate a skeletal pose of the user based on the one or more driving views;

receive a selection of a template mesh for the user;

generate a coarse human mesh representing the user based on the template mesh and the skeletal pose of the user;

construct a UV positional map based on the coarse human mesh;

construct a texture map based on the one or more driving views and the coarse human mesh;

extract a plurality of image features from the texture map, the image features being aligned to the UV positional map and including wrinkles in clothing of the user;

extract a plurality of pose features from the UV positional map, the pose features being aligned to the UV positional map;

generate a plurality of pose-image features based on the UV map-aligned image features and UV map-aligned pose features;

generate textural features and multi-channel volumetric features from the pose-image features, channels of the multi-channel volumetric features corresponding to color channels of the one or the plurality of pixel images of the one or more driving views;

fuse the textural features and the multi-channel volumetric features together; and

render an avatar based on the fused textural features and the multi-channel volumetric features, the avatar including the wrinkles in the clothing of the user.