IP Library › Granted Patent US 12,249,030

Granted Patent B2

US 12,249,030 · App. 17/922,160 · Granted Mar 11, 2025

Generative nonlinear human shape models

Inventors: Cristian Sminchisescu (Kanton of Zurich, CH); Hongyi Xu (Kanton of Zurich, CH); Eduard Gabriel Bazavan (Kanton of Zurich, CH); Andrei Zanfir (Kanton of Zurich, CH); William T. Freeman (Acton, MA); Rahul Sukthankar (Orlando, FL)

Assignee: GOOGLE LLC

G06T17/20 G06N3/0455 G06N3/08 G06T19/20 G06T2219/2021

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12,249,030
App. No.: 17/922,160
Granted: Mar 11, 2025
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

The present disclosure provides a statistical, articulated 3D human shape modeling pipeline within a fully trainable, modular, deep learning framework. In particular, aspects of the present disclosure are directed to a machine-learned 3D human shape model with at least facial and body shape components that are jointly trained end-to-end on a set of training data. Joint training of the model components (e.g., including both facial, hands, and rest of body components) enables improved consistency of synthesis between the generated face and body shapes.

Claims (62)

1. A computer-implemented method to jointly train a machine-learned three-dimensional human shape model in an end-to-end pipeline, the method comprising, for one or more training iterations:

obtaining, by a computing system comprising one or more computing devices, one or more ground truth registered shape scans of a training body, wherein the one or more ground truth registered shape scans of the training body comprise at least a ground truth registered full body scan with an arbitrary pose and a ground truth registered facial detail scan, and wherein the ground truth registered full body scan and the ground truth registered facial detail scan are separate from one another;

encoding, by the computing system using a shape encoder model, an estimated registered full body scan with a resting pose to obtain a rest shape embedding associated with the training body;

decoding, by the computing system using a shape decoder model, the rest shape embedding to obtain identity-based rest shape data for the training body;

encoding, by the computing system using a facial encoder model, data derived from the ground truth registered facial detail scan to obtain a facial expression embedding associated with the training body;

decoding, by the computing system using a facial decoder model, the facial expression embedding to obtain facial expression data for the training body;

generating, by the computing system, a training posed mesh for the training body based at least in part on the identity-based rest shape data, the facial expression data, and a set of pose parameters that correspond to the arbitrary pose;

evaluating a reconstructive loss function that compares the training posed mesh generated for the training body with the ground truth registered full body scan with the arbitrary pose and the ground truth registered facial detail scan, wherein the reconstructive loss function comprises a filter that indicates which vertices of the training posed mesh are compared to the ground truth registered full body scan with the arbitrary pose and which vertices of the training posed mesh are compared to the ground truth registered facial detail scan which is separate from the ground truth registered full body scan with the arbitrary pose;

jointly training the shape encoder model, the shape decoder model, the facial encoder model, and the facial decoder model based at least in part on the reconstructive loss; and

providing the machine-learned three-dimensional human shape model comprising at least the shape decoder model and the facial decoder model.

2. The computer-implemented method of claim 1 , wherein:

generating, by the computing system, the training posed mesh for the training body comprises processing, by the computing system using a pose space deformation model, the set of pose parameters to generate pose-dependent shape adjustments for the training body; and

the pose space deformation model is jointly trained with the shape encoder model, the shape decoder model, the facial encoder model, and the facial decoder model based at least in part on the reconstructive loss.

3. The computer-implemented method of claim 2 , wherein:

generating, by the computing system, the training posed mesh for the training body comprises:

processing, by the computing system using a joint centers prediction model, the identity-based rest shape data to generate a plurality of predicted joint centers for a plurality of joints of a skeleton representation of the training body; and

processing, by the computing system using a blend skinning model, the facial expression data, the pose-dependent shape adjustments, the identity-based rest shape data, and the one or more predicted joint centers to generate the training posed mesh for the training body; and

the joint centers prediction model and the blend skinning model are jointly trained with the shape encoder model, the shape decoder model, the facial encoder model, the facial decoder model, and the pose space deformation model based at least in part on the reconstructive loss.

4. The computer-implemented method of claim 3 , wherein the blend skinning model comprises a linear blend skinning model that has a plurality of learned weights respectively for the plurality of joints.

5. The computer-implemented method of claim 1 , wherein the one or more ground truth registered shape scans of the training body further comprise a ground truth registered hand detail scan, and wherein the reconstructive loss function evaluates a difference between the training posed mesh and the ground truth registered hand detail scan.

6. The computer-implemented method of claim 1 , wherein said jointly training comprises alternating between (1) estimation of the set of pose parameters and (2) updating parameters of the shape encoder model, the shape decoder model, the facial encoder model, and the facial decoder model with the set of pose parameters fixed.

7. The computer-implemented method of claim 1 , wherein the reconstructive loss function evaluates a per-vertex Euclidean distance error with one to one correspondences.

8. A computing system featuring a machine-learned three-dimensional human shape model with at least facial and body shape components jointly trained in an end-to-end pipeline, the computing system comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store a machine-learned three-dimensional human shape model comprising:

a machine-learned facial expression decoder model trained to process a facial expression embedding associated with a human body to generate facial expression data for the human body;

a machine-learned pose space deformation model trained to process a set of pose parameters to generate pose-dependent shape adjustments for the human body; and

a machine-learned shape decoder model trained to process a rest shape embedding associated with the human body to generate identity-based rest shape data for the human body;

wherein the machine-learned three-dimensional human shape model has been trained to generate a posed mesh for the human body based at least in part on the facial expression data, the pose-dependent shape adjustments, and the identity-based rest shape data;

wherein all of the machine-learned facial expression decoder model, the machine-learned pose space deformation model, and the machine-learned shape decoder model have been jointly trained end-to-end based at least in part on a reconstructive loss function that compares a training posed mesh generated by the machine-learned three-dimensional human shape model for a training body with one or more ground truth registered shape scans of the training body;

wherein the one or more ground truth registered shape scans of the training body comprise a ground truth registered full body scan and a ground truth registered facial detail scan that is separate from the ground truth registered full body scan; and

wherein the reconstructive loss function comprises a filter that indicates which vertices of the training posed mesh are compared to the ground truth registered full body scan with the arbitrary pose and which vertices of the training posed mesh are compared to the ground truth registered facial detail scan which is separate from the ground truth registered full body scan with the arbitrary pose.

9. The computing system of claim 8 , wherein the machine-learned three-dimensional human shape model further comprises:

a machine-learned joint centers prediction model trained to process the identity-based rest shape data to generate a plurality of predicted joint centers for a plurality of joints of a skeleton representation of the human body; and

a machine-learned blend skinning model trained to process the facial expression data, the pose-dependent shape adjustments, the identity-based rest shape data, and the one or more predicted joint centers to generate the posed mesh for the human body;

wherein the machine-learned joint centers prediction model and the machine-learned blend skinning model have been jointly trained end-to-end with the machine-learned facial expression decoder model, the machine-learned pose space deformation model, and the machine-learned shape decoder model based at least in part on the reconstruction loss.

10. The computing system of claim 9 , wherein the machine-learned blend skinning model comprises a linear blend skinning model that has a plurality of learned weights respectively for the plurality of joints.

11. The computing system of claim 8 , wherein the one or more ground truth registered shape scans of the training body further comprise a ground truth registered hand detail scan that is separate from the ground truth registered full body scan and the ground truth registered facial detail scan.

12. The computing system of claim 8 , wherein the machine-learned facial expression decoder model was trained as a portion of a facial expression variational autoencoder trained to receive a training head mesh, encode the training head mesh to generate the facial expression embedding, and process the facial expression embedding to generate the facial expression data for the human body.

13. The computing system of claim 8 , wherein the machine-learned shape decoder model was trained as a portion of a shape variational autoencoder trained to receive training body rest shape mesh, encode the training body rest shape mesh to generate the rest shape embedding, and process the rest shape embedding to generate the identity-based rest shape data for the human body.

14. The computing system of claim 8 , wherein the set of pose parameters comprises four feature vectors respectively for head, body, left hand, and right hand joints, and wherein the machine-learned pose space deformation model comprises a rectified linear unit layer configured to jointly receive the four feature vectors and output a single set of outputs.

15. The computing system of claim 8 , wherein the machine-learned three-dimensional human shape model was trained by, for each of a plurality of training examples, alternating between (1) estimation of the set of pose parameters and (2) updating other parameters of the machine-learned three-dimensional human shape model with the set of pose parameters fixed.

16. The computing system of claim 8 , wherein the reconstructive loss function evaluates a per-vertex Euclidean distance error with one to one correspondences.

17. The computing system of claim 8 , wherein the one or more non-transitory computer-readable media further store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:

obtaining a new facial expression embedding, a new set of pose parameters, and a new rest shape embedding associated with a new human body; and

generate a new posed mesh for the new human body based at least in part on the new facial expression embedding, the new set of pose parameters, and the new rest shape embedding.

18. The computing system of claim 17 , wherein obtaining the new facial expression embedding, the new set of pose parameters, and the new rest shape embedding comprises:

generating the new facial expression embedding, the new set of pose parameters, and the new rest shape embedding from a monocular image that depicts the new human body.

19. One or more non-transitory computer-readable media that collectively store a machine-learned three-dimensional human shape model comprising:

a machine-learned facial expression decoder model trained to process a facial expression embedding associated with a human body to generate facial expression data for the human body; and

a machine-learned shape decoder model trained to process a shape embedding associated with the human body to generate identity-based shape data for the human body;

wherein the machine-learned three-dimensional human shape model is trained to generate a posed mesh for the human body based at least in part on the facial expression data, a set of pose parameters, and the identity-based shape data;

wherein the machine-learned facial expression decoder model and the machine-learned shape decoder model have been jointly trained end-to-end based at least in part on a reconstructive loss function that compares a training posed mesh generated by the machine-learned three-dimensional human shape model for a training body with one or more ground truth registered shape scans of the training body;

20. A computer-implemented method to jointly train a machine-learned three-dimensional human shape model in an end-to-end pipeline, the method comprising, for one or more training iterations:

accessing, by the computing system, the machine-learned three-dimensional human shape model comprising one or more body models and one or both of: (i) one or more face models and (ii) one or more hand models;

obtaining, by the computing system comprising one or more computing devices, one or more ground truth registered shape scans of a training body, wherein the one or more ground truth registered shape scans of the training body comprise at least a ground truth registered full body scan with an arbitrary pose and one or both of: (i) a ground truth registered facial detail scan and (ii) a ground truth registered hand detail scan, and wherein the ground truth registered full body scan, the ground truth registered facial detail scan, and ground truth registered hand detail scan are separate from one another;

generating, by the computing system, a training posed mesh for the training body as an output of the machine-learned three-dimensional human shape model;

evaluating a reconstructive loss function that compares the training posed mesh generated for the training body by the machine-learned three-dimensional human shape model with the ground truth registered full body scan with the arbitrary pose and the one or both of the ground truth registered facial detail scan and the ground truth registered hand detail scan, wherein the reconstructive loss function comprises a filter that indicates which vertices of the training posed mesh are compared to the ground truth registered full body scan with the arbitrary pose and which vertices of the training posed mesh are compared to the one or both of the ground truth registered facial detail scan and the ground truth registered hand detail scan which are separate from the ground truth registered full body scan with the arbitrary pose;

jointly training, by the computing system, the one or more body models and the one or both of the one or more face models and the one or more hand models of the machine-learned three-dimensional human shape model based at least in part on the reconstructive loss; and

providing, by the computing system, the machine-learned three-dimensional human shape model as an output.

Assignments (1)

ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 8, 2022

From: SMINCHISESCU, CRISTIAN; XU, HONGYI; BAZAVAN, EDUARD GABRIEL; ZANFIR, ANDREI; FREEMAN, WILLIAM T.; SUKTHANKAR, RAHUL

To: GOOGLE LLC

Reel/Frame 061697/0247 →

Continuity (1)

Related Publication 20230169727A1 · Jun 1, 2023

References Cited (58)

US 20180130245A1 · Kozlov · 2018 [cited by examiner]

US 20190043269A1 · Lin · 2019 [cited by examiner]

US 20190392653A1 · Varanasi · 2019 [cited by examiner]

US 20200058137A1 · Pujades · 2020 [cited by examiner]

Madadi et al; “SMPLR: Deep SMPL reverse for 30 human pose and shape recovery”, arxiv.org, Dec. 27, 2018, XP081457828 (Year: 2018). [cited by examiner]

Pavlakos et al.,“Expressive Body Capture: 3d Hands. Face, and Body from a Single Image”, arXiv:1904.05866vl, Apr. 11, 2019, 22pages (Year: 2019). [cited by examiner]

Romero et al.,“Embodied Hands: Modeling and Capturing Hands and Bodies Together”, ACM Transactions on Graphics, vol. 36, No. 6, Nov. 20, 2017, pp. 1-17, XP058473836. (Year: 2017). [cited by examiner]

Li et aL “Learning a Model of Facial Shape and Expression from 4DS cans”, ACM Transactions on Graphics, vol. 36, No. 6 , Article 194, Nov. 2017, 17pages. (Year: 2017). [cited by examiner]

Hasson et al.,“Leaming Joint Reconstruction of Hands and Manipulated Objects”. arxiv: 1904.05767vl, Apr. 11, 2019. 14pages (Year: 2019). [cited by examiner]

Madadi et al.; SMPLR: Deep SMPL reverse for 3D human pose and shape recovery, arxiv.org, Dec. 27, 2018, XP081457828 (Year: 2018). [cited by examiner]

Allen et al., “The Space of Human Body Shapes: Reconstruction and Parameterization from Range Scans”, ACM Special Interest Group on Computer Graphics and Interactive Techniques, Jul. 27-31, 2003, San Diego, California, … [cited by applicant]

Amberg et al., “Expression Invariant 3D Face Recognition with a Morphable Model”, International Conference on Automatic Face & Gesture Recognition, Sep. 17-19, 2008, 6 pages. [cited by applicant]

Anguelov et al., “Scape: Shape Completion and Animation of People”, ACM Transactions on Graphics, Jul. 2005, 9 pages. [cited by applicant]

Benzine et al., “Deep, Robust and Single Shot 3D Multi-Person Human Pose Estimation from Monocular Images”, arXiv:1911.03391v2, Jan. 7, 2021, 32 pages. [cited by applicant]

Bogo et al., “Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image”, arXiv:1607.08128v1, Jul. 27, 2016, 18 pages. [cited by applicant]

Brunton et al., “Comparative Analysis of Statistical Shape Space”. [cited by applicant]

Cao, “Facewarehouse: A 3D Facial Expression Database for Visual Computing”, Transactions on Visualization and Computer Graphics, vol. 20, No. 3, Mar. 2014, pp. 413-425. [cited by applicant]

CMU Graphics Lab Motion Capture Database, http://mocap.cs.cmu.edu/, 2009. [cited by applicant]

Fieraru et al., “Three-Dimensional Reconstruction of Human Interactions”, Conference on Computer Vision and Pattern Recognition, Seattle, Washington, United States, Jun. 13-19, 2020, pp. 7214-7223. [cited by applicant]

Hasler et al., “A Statistical Model of Human Pose and Body Shape”, Computer Graphics Forum, vol. 28, Issue 2, Mar. 27, 2009, pp. 337-346. [cited by applicant]

Hasson et al., “Learning Joint Reconstruction of Hands and Manipulated Objects”, arXiv:1904.05767v1, Apr. 11, 2019, 14 pages. [cited by applicant]

International Preliminary Report on Patentability for PCT/US2020/030712, mailed on Nov. 10, 2022, 14 pages. [cited by applicant]

International Search Report and Written Opinion for PCT/US2020/03072, mailed on Jan. 21, 2021, 20 pages. [cited by applicant]

Ionescu et al., “Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments”, Transactions on Pattern Analysis and Machine Intelligence, vol. 36, Dec. 11, 2013, pp. 1-15. [cited by applicant]

Jiang et al., “Skeleton-Aware 3D Human Shape Reconstruction From Point Clouds”, International Conference on Computer Vision, Seoul, Korea, Oct. 27-Nov. 2, 2019, pp. 5431-5441. [cited by applicant]

Joo et al., “Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies”, arXiv:1801.01615v1, Jan. 5, 2018, 10 pages. [cited by applicant]

Katircioglu et al., “Learning Latent Representations of 3D Human Pose with Deep Neural Networks”, International Journal of Computer Vision, vol. 126, Jan. 2018, pp. 1326-1341. [cited by applicant]

Kingma et al., “Auto-Encoding Variational Bayes”, arXiv:1312.6114, May 1, 2014, 14 pages. [cited by applicant]

Kolotouros et al., “Learning to Reconstruct 3D Human Pose and Shape Via Model-Fitting in the Loop”, arXiv:1909.12828v1, Sep. 27, 2019, 10 pages. [cited by applicant]

Li et al., “Crowdpose: Efficient Crowded Scenes Pose Estimation and a New Benchmark”, arXiv:1812.00324v2, Jan. 23, 2019, 9 pages. [cited by applicant]

Li et al., “LBS Autoencoder: Self-Supervised Fitting of Articulated Meshes to Point Clouds”, arXiv:1904.10037v1, Apr. 22, 2019, 10 pages. [cited by applicant]

Li et al., “Learning a Model of Facial Shape and Expression from 4D Scans”, ACM Transactions on Graphics, vol. 36, No. 6, Article 194, Nov. 2017, 17 pages. [cited by applicant]

Lin et al., “Microsoft COCO: Common Objects in Context”, arXiv:1405.0312v3, Feb. 21, 2015, 15 pages. [cited by applicant]

Liu et al., “Markerless Motion Capture of Interacting Characters Using Multi-View Image Segmentation”, Computer Vision and Pattern Recognition, Colorado Springs, Colorado, United States, Jun. 20-25, 2011, pp. 1249-1256. [cited by applicant]

Loper et al., “SMPL: A Skinned Multi-Person Linear Model”, ACM Transactions on Graphics, vol. 34, Issue 6, Article 248, Nov. 2015, pp. 1-16. [cited by applicant]

Luvizon et al., “2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning”, arXiv:1802.09232v2, Mar. 2, 2018, 12 pages. [cited by applicant]

Madadi et al., “SMPLR: Deep SMPL Reverse for 3D Human Pose and Shape Recovery”, arXiv:1812.10766v2, Aug. 8, 2019, 11 pages. [cited by applicant]

Mahmood et al., “AMASS: Archive of Motion Capture as Surface Shapes”, arXiv:1904.03278v1, Apr. 5, 2019, 12 pages. [cited by applicant]

Martinez et al., “A Simple Yet Effective Baseline for 3D Human Pose Estimation”, arXiv:1705.03098v2, Aug. 4, 2017, 10 pages. [cited by applicant]

Mehta et al., “Single-Shot Multi-Person 3D Pose Estimation from Monocular RGB”, arXiv:1712.03453v3, Aug. 28, 2018, 16 pages. [cited by applicant]

Mehta et al., “VNect: Real-Time 3D Human Pose Estimation with a Single RGB Camera”, arXiv:1705.01583v1, May 3, 2017, 13 pages. [cited by applicant]

Oberweger et al., “Training a Feedback Loop for Hand Pose Estimation”, arXiv:1609.09698v1, Sep. 30, 2016, 9 pages. [cited by applicant]

Oikonomidis et al., “Efficient Model-Based 3d Tracking of Hand Articulations Using Kinect”, British Machine Vision Conference, Sep. 2011, pp. 1-11. [cited by applicant]

Pavlakos et al., “Expressive Body Capture: 3d Hands, Face, and Body from a Single Image”, arXiv:1904.05866v1, Apr. 11, 2019, 22 pages. [cited by applicant]

Ploumpis et al., “Combining 3D Morphable Models: A Large Scale Face-And-Head Model”, Conference on Computer Vision and Pattern Recognition, Long Beach, California, United States, Mar. 11, 2019, pp. 10934-10943. [cited by applicant]

Popa et al., “Deep Multitask Architecture for Integrated 2D and 3D Human Sensing”, arXiv:1701.08985v1, Jan. 31, 2017, 10 pages. [cited by applicant]

Rhodin et al., “General Automatic Human Shape and Motion Capture Using Volumetric Contour Cues”, arXiv:1607.08659v2, Oct. 21, 2016, 19 pages. [cited by applicant]

Romero et al., “Embodied Hands: Modeling and Capturing Hands and Bodies Together”, arXiv:2201.02610v1, Jan. 7, 2022, 19 pages. [cited by applicant]

Su et al., “Multi-Person Pose Estimation with Enhanced Channel-wise and Spatial Information”, arXiv:1905.03466v1, May 9, 2019, 9 pages. [cited by applicant]

Taylor et al., “Efficient and Precise Interactive Hand Tracking Through Joint, Continuous Optimization of Pose and Correspondences”, Special Interest Group on Computer Graphics and Interactive Techniques Conference, Ana… [cited by applicant]

Tzionas et al., “Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation”, arXiv:1506.02178v4, Mar. 7, 2016, 24 pages. [cited by applicant]

Xu et al., “GHUM & GHUML: Generative 3D Human Shape and Articulated Pose Models”, Conference on Computer Vision and Pattern Recognition, Seattle, Washington, United States, Jun. 13-19, 2020, pp. 6184-6193. [cited by applicant]

Yang et al., “3D Human Pose Estimation in the Wild by Adversarial Learning”, arXiv:1803.09722v2, Apr. 16, 2018, 10 pages. [cited by applicant]

Yang et al., “Expression Flow for 3D-Aware Face Component Transfer”, Graphics, vol. 30, Issue 4., Article No. 60, 10 pages. [cited by applicant]

Yoshiyasu et al., “As-Conformal-As-Possible Surface Registration”, Computer Graphics Forum, vol. 33, No. 5, Jul. 9, 2014, 11 pages. [cited by applicant]

Zanfir et al., “Deep Network for the Integrated 3D Sensing of Multiple People in Natural Images”, Conference on Neural Information Processing Systems, Montreal, Quebec, Canada, Dec. 3-8, 2018, 10 pages. [cited by applicant]

Zanfir et al., “Monocular 3D Pose and Shape Estimation of Multiple People in Natural Scenes—The Importance of Multiple Scene Constraints”, Conference on Computer Vision and Pattern Recognition, Salt Lake City, Utah, Uni… [cited by applicant]

Zanfir et al., “Weakly Supervised 3d Human Pose and Shape Reconstruction with Normalizing Flows”, arXiv:2003.10350v2, Aug. 22, 2020, 20 pages. [cited by applicant]