Machine learning acceleration of complex deformations such as muscle, skin and clothing simulation
Methods and systems are provided preparing training data for training a neural network to simulate deformations of a surface of a CG character, for training a neural network to simulate deformations of a surface of a CG character and for employing a neural network to simulate deformations of a surface of a CG character. Matrix decomposition techniques are used to generate the training data and are subsequently used by trained neural networks during inference to reconstruct CG character surfaces. The inference methods and systems are suitable for real time animation applications.
1. A method for preparing training data for training a neural network to simulate deformations of a surface of a CG character, the method comprising:
obtaining a distribution of joint angles and/or bone positions of a CG character over a set of animation data comprising a plurality of frames;
randomly generating a plurality of random poses according to the distribution of joint angles and/or bone positions;
generating a high-fidelity deformation of the surface of the CG character for each of the plurality of random poses;
transforming each of the high-fidelity deformations from a respective pose coordinate system to a rest pose coordinate system to obtain a plurality of warped rest poses, each warped rest pose corresponding to one of the high-fidelity deformations and one of the random poses and each warped rest pose parameterized at least in part by a three-dimensional (3D) surface mesh comprising a plurality of vertices;
parsing the warped rest poses into a plurality of regions to obtain, for each warped rest pose, a corresponding plurality of warped rest regions, each warped rest region parameterized at least in part by a regional three-dimensional (3D) surface mesh comprising a regional plurality of vertices from among the plurality of vertices of the vertices of the warped rest pose;
determining an approximation weight for each regional vertex of each of the plurality of warped rest regions;
for each region:
decomposing the warped rest region over the plurality of warped rest poses to obtain, for the region: a regional decomposition neutral vector, a regional set of decomposition basis (blendshape) vectors and, for each warped rest pose, a regional set of decomposition weights;
wherein, for each warped rest pose, the corresponding regional set of decomposition weights together with the regional decomposition neutral vector and the regional set of decomposition basis (blendshape) vectors can be used to at least approximately reconstruct the warped rest region of the warped rest pose;
wherein decomposing the warped rest region over the plurality of warped rest poses is based at least in part on the approximation weights; and
determining the training data to comprise the plurality of random poses and, for each random pose and for each region, the corresponding set of decomposition weights.
2. A method according to claim 1 wherein determining the approximation weight for each regional vertex of each of the plurality of warped rest regions comprises, for each regional vertex of each of the plurality of warped rest regions, determining a derivative of a position of the regional vertex in the warped rest region with respect to a position of the vertex in the corresponding high-fidelity deformation.
3. A method according to claim 2 wherein the approximation weight for each regional vertex of each of the plurality of warped rest regions is inversely proportional to the determined derivative of the position of the regional vertex in the warped rest region with respect to a position of the vertex in the corresponding high-fidelity deformation.
4. A method according to claim 1 wherein, for each region, decomposing the warped rest region over the plurality of warped rest poses comprises determining a regional basis size (e.g. a number of regional decomposition basis (blendshape) vectors in the set of regional decomposition basis (blendshape) vectors).
5. A method according to claim 4 wherein determining the basis size comprises performing one or more principal component analysis (PCA) decompositions of the warped rest region over the plurality of warped rest poses and determining the basis size based on the one or more PCA decompositions.
6. A method according to claim 1 wherein decomposing the warped rest region over the plurality of warped rest poses comprises performing a weighted low rank approximation (WLRA) decomposition based on the warped rest region over the plurality of warped rest poses and the approximation weights.
7. A method according to claim 1 wherein at least some of the warped rest regions share at least some vertices from among their respective regional pluralities of vertices.
8. A method according to claim 7 wherein parsing the warped rest poses into the plurality of regions comprises assigning per-region weights to vertices that are shared between warped rest regions.
9. A method according to claim 1 wherein:
each random pose is parameterized at least in part by a number of bones (num_bones) and, for each bone, a set of bone matrix components, the num-bones sets of bone matrix components together characterizing the joint angles and/or bone positions of the random pose; and
parsing the warped rest poses into the plurality of regions comprises:
determining a correlation between each coordinate of each vertex of each warped rest pose and each bone matrix component over the plurality of warped rest poses;
clustering the vertices into n clusters based at least in part on the determined correlations; and
dilating the n clusters to determine n warped rest regions, one warped rest region corresponding to each cluster, wherein the regional plurality of vertices for each warped rest region comprises the vertices in its corresponding cluster and one or more vertices from neighboring clusters; and
assigning per-region weights to vertices that belong to more than one warped rest region.
10. A method according to claim 9 wherein dilating the n clusters to determine n warped rest regions comprises, for each warped rest region, determining the one or more vertices from neighboring clusters to be within a threshold distance metric from at least one of the vertices in the cluster corresponding to the warped rest region.
11. A method according to claim 10 wherein assigning per-region weights to vertices that belong to more than one warped rest region comprises, for each particular vertex belonging to more that one warped rest region, performing an averaging process of the per-region weights of other vertices within the threshold distance metric from the particular vertex.
12. A method according to claim 11 wherein performing the averaging process comprises performing a weighted averaging process, wherein weights for the weighted averaging process are determined at least in part on distance metrics of the other vertices relative to the particular vertex.
13. A method according to claim 1 wherein the animation data comprises the joint angles and/or bone positions.
14. A method according to claim 1 wherein obtaining the distribution of joint angles and/or bone positions of the CG character over the set of animation data comprises: receiving the set of animation data of the CG character; and determining the distribution of joint angles and/or bone positions of the CG character in each frame of the set of animation data.
15. A method according to claim 14 wherein the distribution of joint angles and/or bone positions of the character in the animation comprises a multivariate Gaussian distribution of the joint angles and/or bone positions of the CG character over the plurality of frames of the set of animation data.
16. A method according to claim 1 wherein the joint angles and/or bone positions are parameterized at least in part by a plurality of bones and, for each bone, a corresponding bone matrix.
17. A method according to claim 1 wherein each random pose is parameterized at least in part by a plurality of bones and, for each bone, a bone matrix comprising a set of bone matrix components, the plurality of sets of bone matrix components together characterizing the joint angles and/or bone position of the random pose.
18. A method for preparing training data for training a neural network to simulate deformations of a surface of a CG character, the method comprising:
obtaining a distribution of joint angles and/or bone positions of a CG character over a set of animation data comprising a plurality of frames;
randomly generating a plurality of random poses according to the distribution of joint angles and/or bone positions;
generating a high-fidelity deformation of the surface of the CG character for each of the plurality of random poses;
transforming each of the high-fidelity deformations from a respective pose coordinate system to a rest pose coordinate system to obtain a plurality of warped rest poses, each warped rest pose corresponding to one of the high-fidelity deformations and one of the random poses and each warped rest pose parameterized at least in part by a three-dimensional (3D) surface mesh comprising a plurality of vertices;
determining an approximation weight for each vertex of each of the plurality of warped rest poses;
decomposing the plurality of warped rest poses to obtain: a decomposition neutral vector, a set of decomposition basis (blendshape) vectors and, for each warped rest pose, a set of decomposition weights;
wherein, for each warped rest pose, the corresponding set of decomposition weights together with the decomposition neutral vector and the set of decomposition basis (blendshape) vectors can be used to at least approximately reconstruct the warped rest pose;
wherein decomposing the plurality of warped rest poses is based at least in part on the approximation weights; and
determining the training data to comprise the plurality of random poses and, for each random pose, the corresponding set of decomposition weights.
19. A method for training a neural network to simulate deformations of a surface of a CG character, the method comprising:
(a) receiving training data comprising training poses and, for each training pose, a plurality of sets of training blendshape weights, each set of blendshape weights corresponding to a region of the surface of the CG character; and
for each region:
(b) employing an untrained or partially trained neural network comprising a plurality of trainable parameters to predict blendshape weights based on one of the plurality of training poses;
(c) determining an error metric, the error metric based at least in part on the predicted blendshape weights and the set of training blendshape weights corresponding to the region and the one of the plurality of training poses;
(d) updating the trainable parameters of the neural network based at least in part on the error metric;
(e) evaluating a loop exit criteria and:
if the loop exit criteria is satisfied, proceeding to step (f); or
if the loop exit criteria is not satisfied, repeating steps (b), (c), (d) and (e) with a different one of the plurality of training poses; and
(f) parameterizing the trained neural network based on at least in part on the updated trainable parameters after the last iteration of step (d).
20. A method according to claim 19 wherein the training data is obtained by the method of claim 1 .