Methods and systems of generating an anatomically-constrained local model for performance capture
View Patent ↗Techniques and systems are described for generating an anatomically-constrained local model and for performing performance capture using the model. The local model includes a local shape subspace and an anatomical subspace. In one example, the local shape subspace constrains local deformation of various patches that represent the geometry of a subject's face. In the same example, the anatomical subspace includes an anatomical bone structure, and can be used to constrain movement and deformation of the patches globally on the subject's face. The anatomically-constrained local face model and performance capture technique can be used to track three-dimensional faces or other parts of a subject from motion data in a high-quality manner. Local model parameters that best describe the observed motion of the subject's physical deformations (e.g., facial expressions) under the given constraints are estimated through optimization. The optimization can solve for rigid local patch motion, local patch deformation, and the rigid motion of the anatomical bones. The solution can be formulated as an energy minimization problem for each frame that is obtained for performance capture.
1. A computer-implemented method of generating an anatomically-constrained model of a face of a subject, the method comprising:
obtaining one or more three-dimensional shapes of the face of the subject;
segmenting the face into a plurality of patches;
determining a plurality of local shape subspaces for the plurality of patches, each local shape subspace including local shape deformation of a corresponding patch, wherein a local shape subspace for a patch includes a plurality of deformation shapes for the patch, and wherein a deformation shape of the patch defines a deformation of the patch for a facial expression;
determining an anatomical subspace, the anatomical subspace including tissue thickness constraints for the plurality of patches and an anatomical bone structure, the anatomical subspace globally constraining local shape deformation and tissue thickness for the plurality of patches, wherein the patch is constrained by the anatomical bone structure using one or more tissue thickness constraints determined for the plurality of deformation shapes of the patch; and
generating the anatomically-constrained model of the face by combining the plurality of local shape subspaces and the anatomical subspace.
2. The method of claim 1 , wherein determining the plurality of local shape subspaces includes determining the plurality of deformation shapes by removing rigid motion from facial expressions included in the obtained one or more three-dimensional shapes.
3. The method of claim 2 , further comprising:
aligning a deformed patch of a facial expression in an obtained three-dimensional shape with a corresponding neutral patch of a neutral facial expression to obtain an aligned patch; and
subtracting a shape of the corresponding neutral patch from the aligned patch to obtain a deformation shape for the deformed patch.
4. The method of claim 1 , wherein the anatomical bone structure constrains deformation and movement of each of the plurality of patches.
5. The method of claim 1 , further comprising performing facial performance tracking of the subject using the anatomically-constrained model of the face.
6. The method of claim 5 , wherein the facial performance tracking is performed using input from a single camera.
7. The method of claim 1 , wherein the anatomical bone structure includes at least one or more of a skull or a jawbone.
8. A system for generating an anatomically-constrained model of a face of a subject, comprising:
a memory storing a plurality of instructions; and
one or more processors configurable to:
obtain one or more three-dimensional shapes of the face of the subject;
segment the face into a plurality of patches;
determine a plurality of local shape subspaces for the plurality of patches, each local shape subspace including local shape deformation of a corresponding patch, wherein a local shape subspace for a patch includes a plurality of deformation shapes for the patch, and wherein a deformation shape of the patch defines a deformation of the patch for a facial expression;
determine an anatomical subspace, the anatomical subspace including tissue thickness constraints for the plurality of patches and an anatomical bone structure, the anatomical subspace globally constraining local shape deformation and tissue thickness for the plurality of patches, wherein the patch is constrained by the anatomical bone structure using one or more tissue thickness constraints determined for the plurality of deformation shapes of the patch; and
generate the anatomically-constrained model of the face by combining the plurality of local shape subspaces and the anatomical subspace.
9. The system of claim 8 , wherein determining the plurality of local shape subspaces includes determining the plurality of deformation shapes by removing rigid motion from facial expressions included in the obtained one or more three-dimensional shapes.
10. The system of claim 9 , wherein the one or more processors are configurable to:
align a deformed patch of a facial expression in an obtained three-dimensional shape with a corresponding neutral patch of a neutral facial expression to obtain an aligned patch; and
subtract a shape of the corresponding neutral patch from the aligned patch to obtain a deformation shape for the deformed patch.
11. The system of claim 8 , wherein the anatomical bone structure constrains deformation and movement of each of the plurality of patches.
12. The system of claim 8 , wherein the one or more processors are configurable to perform facial performance tracking of the subject using the anatomically-constrained model of the face.
13. The system of claim 12 , wherein the facial performance tracking is performed using input from a single camera.
14. The system of claim 8 , wherein the anatomical bone structure includes at least one or more of a skull or a jawbone.
15. A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising:
instructions that cause the one or more processors to obtain one or more three-dimensional shapes of a face of a subject;
instructions that cause the one or more processors to segment the face into a plurality of patches;
instructions that cause the one or more processors to determine a plurality of local shape subspaces for the plurality of patches, each local shape subspace including local shape deformation of a corresponding patch, wherein a local shape subspace for a patch includes a plurality of deformation shapes for the patch, and wherein a deformation shape of the patch defines a deformation of the patch for a facial expression;
instructions that cause the one or more processors to determine an anatomical subspace, the anatomical subspace including tissue thickness constraints for the plurality of patches and an anatomical bone structure, the anatomical subspace globally constraining local shape deformation and tissue thickness for the plurality of patches, wherein the patch is constrained by the anatomical bone structure using one or more tissue thickness constraints determined for the plurality of deformation shapes of the patch; and
instructions that cause the one or more processors to generate the anatomically-constrained model of the face by combining the plurality of local shape subspaces and the anatomical subspace.
16. The non-transitory computer-readable memory of claim 15 , wherein determining the plurality of local shape subspaces includes determining the plurality of deformation shapes by removing rigid motion from facial expressions included in the obtained one or more three-dimensional shapes.
17. The non-transitory computer-readable memory of claim 16 , further comprising:
instructions that cause the one or more processors to align a deformed patch of a facial expression in an obtained three-dimensional shape with a corresponding neutral patch of a neutral facial expression to obtain an aligned patch; and
instructions that cause the one or more processors to subtract a shape of the corresponding neutral patch from the aligned patch to obtain a deformation shape for the deformed patch.
18. The non-transitory computer-readable memory of claim 15 , wherein the anatomical bone structure constrains deformation and movement of each of the plurality of patches, the anatomical bone structure including at least one or more of a skull or a jawbone.
19. The non-transitory computer-readable memory of claim 15 , further comprising instructions that cause the one or more processors to perform facial performance tracking of the subject using the anatomically-constrained model of the face, the facial performance tracking being performed using input from a single camera.
20. A computer-implemented method of generating an anatomically-constrained model of a subject, the method comprising:
obtaining one or more three-dimensional shapes of a part of the subject;
segmenting the part into a plurality of patches;
determining a plurality of local shape subspaces for the plurality of patches, each local subspace including local shape deformation of a corresponding patch, wherein a local shape subspace for a patch includes a plurality of deformation shapes for the patch, and wherein a deformation shape of the patch defines a deformation of the patch for an observed shape of the part;
determining an anatomical subspace, the anatomical subspace including material thickness constraints for the plurality of patches and a rigid structure, the anatomical subspace globally constraining local shape deformation and material thickness for the plurality of patches, wherein the patch is constrained by the rigid structure using one or more material thickness constraints determined for the plurality of deformation shapes of the patch; and
generating the anatomically-constrained model of the part by combining the plurality of local shape subspaces and the anatomical subspace.