Skeletal systems for animating virtual avatars
Skinning parameters used to animate a virtual avatar can include mesh weights and joint transforms of a skeleton. Systems and methods are provided for determining skinning parameters using an optimization process subject to constraints based on human-understandable or anatomically-motivated relationships among skeletal joints. Input to the optimization process can include a high-order skeleton and the applied constraints can dynamically change during the optimization. The skinning parameters can be used in linear blend skinning (LBS) applications in augmented reality.
1. A system comprising:
non-transitory computer storage configured to store:
a target mesh representing a target pose of a virtual avatar, the target mesh comprising a plurality of target mesh vertices;
an initialization mesh representing an initial pose of the virtual avatar, the initialization mesh comprising a plurality of initialization mesh vertices and an initialization skeleton comprising a plurality of joints;
skinning parameters representing deformation of the initialization mesh responsive to transformation of the initialization skeleton into the target pose of the virtual avatar; and
a set of constraints associated with the skinning parameters or the joints, wherein the set of constraints comprises a constraint incorporating a segmentation set that relates a set of one or more initialization mesh vertices to a set of one or more joints of the initialization skeleton; and
a hardware processor in communication with the non-transitory computer storage, the hardware processor programmed to:
access the target mesh, the initialization mesh, the initialization skeleton, and the set of constraints;
perform a constrained optimization process for the set of one or more initialization mesh vertices in the segmentation set to determine the skinning parameters that provide a match to the target mesh, the match being determined responsive to a difference between 1) positions of the set of one or more initialization mesh vertices in the segmentation set adjusted as a result of a transformation of the initialization skeleton into the target pose and 2) positions of a corresponding set of one or more target mesh vertices satisfying an error threshold, the constrained optimization process subject to the set of constraints; and
output the skinning parameters.
2. The system of claim 1 , wherein the target mesh is determined from photographic scans of a human or animal subject performing a pose.
3. The system of claim 1 , wherein the target mesh is determined from artist-driven sculpting, a computational simulation of the target pose, or a biometrically-derived model.
4. The system of claim 1 , wherein the initialization skeleton comprises a high-order skeleton.
5. The system of claim 1 , wherein the initialization skeleton comprises a low-order skeleton in a base or neutral pose.
6. The system of claim 1 , wherein the initialization skeleton further comprises a skin cluster mapping of vertices to joints of a skeleton.
7. The system of claim 1 , wherein the skinning parameters comprise a weight map that measures amount of influence of a joint on a vertex.
8. The system of claim 1 , wherein the skinning parameters comprise six degree of freedom parameters associated with movement of each joint.
9. The system of claim 8 , wherein the six degree of freedom parameters comprise a rotation and a translation.
10. The system of claim 1 , wherein the hardware processor is further programmed to calculate a radial basis function (RBF) representation associated with the skinning parameters.
11. The system of claim 1 , wherein the constrained optimization process is programmed to implement a falloff configured to blend a result from a first region into a second region.
12. The system of claim 1 , wherein the constrained optimization process comprises linear or quadratic programming.
13. The system of claim 1 , wherein the constrained optimization process comprises a nonlinear solver utilizing first derivative or second derivative estimates.
14. The system of claim 1 , wherein the set of constraints comprises one or more of:
a non-negativity constraint for weights of a weight map;
a normalization constraint for the total influence of all joints on a vertex;
a maximum number of non-zero weights for any vertex;
a constraint that a rotation parameter associated with joint movement represents a three-dimensional rotation; or
a constraint that a rotation parameter associated with joint movement comprises an orthogonal matrix.
15. The system of claim 1 , wherein the set of constraints comprises one or more of:
a constraint that restricts a child joint to being a fixed distance from a parent joint;
a smoothness constraint that restricts neighboring vertices of a mesh to have similar weights in a weigh map;
a geometric constraint representing an anatomical characteristic of a joint;
a geometric constraint representing a length, a lateral size, or a range of movement (ROM) of a joint;
a constraint that locks a subset of joints from being updated during the constrained optimization process;
a constraint that a joint position can translate only along a specific axis;
a constraint that locks a joint characteristic during the constrained optimization process; or
a constraint that sets a weight for a vertex to zero if a joint is located beyond a threshold distance from the vertex.
16. The system of claim 1 , wherein the hardware processor is programmed to dynamically update the set of constraints during the constrained optimization process.
17. The system of claim 1 , wherein the constrained optimization process is performed only for the set of one or more initialization mesh vertices in the segmentation set.
18. A method comprising:
under control of a hardware processor in communication with non-transitory storage:
accessing a target mesh representing a target pose for a virtual avatar;
accessing an initialization mesh for use in matching the target mesh;
accessing an initialization skeleton comprising a plurality of joints;
accessing a set of constraints comprising a constraint for a segmentation set that relates a set of one or more vertices of the initialization mesh to a set of one or more joints of the initialization skeleton;
performing constrained optimization to determine skinning parameters that represent the target pose, the constrained optimization performed subject to the set of constraints, the constrained optimization initialized with the initialization mesh, the constrained optimization comprising:
determining a match of the initialization mesh deformed by the skinning parameters as a result of a transformation of the initialization skeleton into the target pose with the target mesh responsive to a difference between 1) positions of the set of one or more vertices of the initialization mesh in the segmentation set adjusted by the skinning parameters and 2) positions of a corresponding set of one or more vertices of the initialization mesh satisfying an error threshold; and
outputting the skinning parameters for which the match has been determined.
19. The method of claim 18 , wherein the constrained optimization is performed only for the set of one or more mesh vertices of the initialization mesh in the segmentation set.
20. The method of claim 18 , wherein the set of constraints comprises one or more of:
a non-negativity constraint for weights of a weight map;
a normalization constraint for the total influence of all joints on a vertex;
a maximum number of non-zero weights for any vertex;
a constraint that a rotation parameter associated with joint movement represents a three-dimensional rotation; or
a constraint that a rotation parameter associated with joint movement comprises an orthogonal matrix.
21. The method of claim 18 , wherein the set of constraints comprises one or more of:
a constraint that restricts a child joint to being a fixed distance from a parent joint;
a smoothness constraint that restricts neighboring vertices of a mesh to have similar weights in a weigh map;
a geometric constraint representing an anatomical characteristic of a joint;
a geometric constraint representing a length, a lateral size, or a range of movement (ROM) of a joint;
a constraint that locks a subset of joints from being updated during the constrained optimization;
a constraint that a joint position can translate only along a specific axis;
a constraint that locks a joint characteristic during the constrained optimization; or
a constraint that sets a weight for a vertex to zero if a joint is located beyond a threshold distance from the vertex.
22. The method of claim 18 , wherein the initialization skeleton comprises a high-order skeleton.