IP Library › Granted Patent US 12,374,046
Granted Patent B2
US 12,374,046 · App. 18/179,415 · Granted Jul 29, 2025

Device and method for estimating the shape and pose of a body

Inventors: Xiangyu Xu (Singapore, SG); Lijuan Liu (Singapore, SG); Shuicheng Yan (Singapore, SG)
Assignee: Shopee IP Singapore Private Limited
G06T17/20G06T7/50G06T7/70G06T7/80G06T2207/20081G06T2207/20084G06T2207/30196
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,374,046
App. No.
18/179,415
Granted
Jul 29, 2025
Kind
B2
Abstract

A method for estimating the shape and pose of a body includes obtaining an image of the body, extracting image features, and transforming the image features to a set of parameter values describing the body's shape and pose using a transformer comprising a cascade of transformer units, wherein each transformer unit operates on a representation of a respective version of the set of parameter values and modifies the representation of the respective version of the set of parameter values by the image features weighted by an attention of the respective version of the set of parameter values to the image features in sequence with a self-attention layer modifying the representation of the respective version of the set of parameter values by components of the representation of parameter values weighted by an attention between the components of the representation of the respective version of the set of parameter values to themselves.

Claims (40)

1. A method for estimating, using a neural network, the shape and pose of a body, the method comprising:

obtaining an image of the body;

extracting image features from the image;

transforming the image features to a set of parameter values describing the shape and pose of the body using a transformer comprising a cascade of transformer units,

wherein each transformer unit operates on a representation of a respective version of the set of parameter values and comprises

a cross-attention layer modifying the representation of the respective version of the set of parameter values by the image features weighted by an attention of the respective version of the set of parameter values to the image features;

and

a self-attention layer modifying the representation of the respective version of the set of parameter values by components of the representation of the respective version of the set of parameter values weighted by an attention among the components of the representation of the respective version of the set of parameter values

and processing the representation of the respective version of the set of parameter values by the cross-attention layer followed by the self-attention layer.

2. The method of claim 1 , wherein the self-attention layer comprises at least one self-attention unit performing respective attention operation with query, key and value, for which each of the query, the key and the value are set to, up to a modification by positional encoding, the representation of the respective version of the set of parameter values.

3. The method of claim 1 , wherein the cross-attention layer comprises at least one cross-attention unit performing respective attention operation with query, key and value, for which the query is set to, up to a modification by positional encoding, the representation of the respective version of the set of parameter values and the key and the value are set to, up to a modification by positional encoding, the image features or a subset of the image features.

4. The method of claim 1 , wherein the image features comprise image features for each of a plurality of scales and wherein the cross-attention layer comprises a cross-attention unit for each layer modifying the representation of the respective version of the set of parameter values by an attention from the representation to image features of the scale.

5. The method of claim 4 , wherein the cross-attention layer outputs a modified version of the representation of the set of parameter values which it generates by averaging results of the modifying performed by the cross-attention units for the scales.

6. The method of claim 1 , wherein the set of parameters comprises values of parameters of a parametric body model.

7. The method of claim 6 , wherein the method further comprises generating a mesh representation of the body from the values of the parameters of the parametric body model.

8. The method of claim 1 , wherein the set of parameter values comprises values for a set of parameters including rotation parameters for each of a plurality of parts of the body.

9. The method of claim 8 , wherein the cross-attention layer comprises a local attention unit modifying the representation of the respective version of the set of parameter values by, for each of the rotation parameters, the image features of a patch showing the body part weighted by an attention of the respective version of the set of parameter values to the image features of the patch showing the body part.

10. The method of claim 9 , wherein the cross-attention layer outputs a modified version of the representation of the set of parameter values which it generates by summing a result of the modifying performed by the local attention unit and a result of the modifying of the representation of the respective version of the set of parameter values by the image features weighted by an attention of the respective version of the set of parameter values to the image features.

11. The method of claim 1 , wherein the set of parameter values comprises values for a set of shape parameters.

12. The method of claim 1 , wherein the image is obtained using a camera and the set of parameter values comprises values for a set of camera parameters.

13. The method of claim 1 , comprising extracting the image features using a convolutional neural network.

14. The method of claim 1 , comprising training the transformer by supervised learning.

15. The method of claim 14 , wherein the transformer is trained using a ground truth for at least one of 2D vertices of a model of the body and joint configurations of the body.

16. The method of claim 1 , wherein the cross-attention layer disregards attention of the representation of the respective version of the set of parameter values to itself and disregards attention of the respective version of the set of parameter values to the image features.

17. The method of claim 1 , comprising generating an initial version of the set of parameter values from the extracted image features and generating a sequence of successively updated versions of the set of parameter values by generating each updated version of the set of parameter values by updating a previous version of the set of parameter values in the sequence according to a result of modifying a representation of the previous version of the set of parameter values by one or more of the transformer units and mapping a result of the modifying of the representation of the previous version of the set of parameter values to a corresponding version of the set of parameter values.

18. A data processing device for estimating the shape and pose of a body using a neural network, the data processing device configured to:

obtain an image of the body;

extract image features from the image;

transform the image features to a set of parameter values describing the shape and pose of the body using a transformer comprising a cascade of transformer units,

wherein each transformer unit operates on a representation of a respective version of the set of parameter values and comprises

a cross-attention layer modifying the representation of the respective version of the set of parameter values by the image features weighted by an attention of the respective version of the set of parameter values to the image features; and

a self-attention layer modifying the representation of the respective version of the set of parameter values by components of the representation of the respective version of the set of parameter values weighted by an attention among the components of the representation of the respective version of the set of parameter values and is configured to process the representation of the respective version of the set of parameter values by the cross-attention layer followed by the self-attention layer.

19. A non-transitory computer-readable medium, comprising instructions which, when executed, cause one or more processors to:

estimate, based on a neural network, the shape and pose of a body, wherein the instructions that cause the one or more processors to estimate also cause the one or more processors to:

obtain an image of the body;

extract image features from the image;

transform the image features to a set of parameter values describing the shape and pose of the body using a transformer comprising a cascade of transformer units,

wherein each transformer unit operates on a representation of a respective version of the set of parameter values and comprises

a cross-attention layer modifying the representation of the respective version of the set of parameter values by the image features weighted by an attention of the respective version of the set of parameter values to the image features; and

a self-attention layer modifying the representation of the respective version of the set of parameter values by components of the representation of the respective version of the set of parameter values weighted by an attention among the components of the representation of the respective version of the set of parameter values and process the representation of the respective version of the set of parameter values by the cross-attention layer followed by the self-attention layer.

Assignments (2)
NUNC PRO TUNC ASSIGNMENT Recorded Jun 19, 2025
From: GARENA ONLINE PRIVATE LIMITED
To: SHOPEE IP SINGAPORE PRIVATE LIMITED
Reel/Frame 071453/0969 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 13, 2023
From: XU, XIANGYU; LIU, LIJUAN; YAN, SHUICHENG
To: GARENA ONLINE PRIVATE LIMITED
Reel/Frame 062955/0647 →
Priority Claims (1)
SG 10202202251U · Mar 7, 2022 · national
Continuity (1)
Related Publication 20230281926A1 · Sep 7, 2023
References Cited (89)
US 11887248B2 · Zakharov · 2024 [cited by examiner]
US 11902705B2 · Shih · 2024 [cited by examiner]
US 20210158107A1 · Karanam et al. · 2021 [cited by applicant]
US 20220292654A1 · Zhang · 2022 [cited by examiner]
US 20230267285A1 · Wu · 2023 [cited by examiner]
CN 112232134A · 2021 [cited by applicant]
CN 112907736A · 2021 [cited by applicant]
CN 113936333A · 2022 [cited by applicant]
Stefania Cristina “The Transformer Attention Mechanism”, Jan. 6, 2023, Making developers awesome at machine learning, pp. 1-3 (Year: 2023). [cited by examiner]
The Transformer Attention Mechanism (Year: 2023). [cited by examiner]
Liu, Ze et al., “Swin Transformer: Hierarchical Vision Transformer using Shifted Windows”, International Conference on Computer Vision (ICCV), 2021, pp. 10012-10022. [cited by applicant]
Luo, Zhengyi, “3D Human Motion Estimation via Motion Compression and Refinement”, Asian Conference on Computer Vision (ACCV), Oct. 5, 2020, 21 pages. [cited by applicant]
Mehta, Dushyant et al., “Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB”, International Conference on 3D Vision (3DV 2018), 2018, 16 pages. [cited by applicant]
Xu, Xiangyu et al., “3D Human Shape and Pose from a Single Low-Resolution Image with Self-Supervised Learning”, arXiv, Aug. 9, 2020, 17 pages. [cited by applicant]
Moreno-Noguer, Francesc, “3D Human Pose Estimation from a Single Image via Distance Matrix Regression”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 10 pages. [cited by applicant]
Nah, Seungjun et al., “Deep multi-scale convolutional neural network for dynamic scene deblurring”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, 21 pages. [cited by applicant]
Omran, Mohamed et al., “Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose and Shape Estimation”, International Conference on 3D Vision (3DV 2018), 2018, 13 pages. [cited by applicant]
Pavlakos, Georgios et al., “Learning to Estimate 3D Human Pose and Shape from a Single Color Image”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, 10 pages. [cited by applicant]
Romero, Javier et al., “Embodied Hands: Modeling and Capturing Hands and Bodies Together”, ACM Transactions on Graphics vol. 36, No. 6, Nov. 2017, 19 pages. [cited by applicant]
Schönemann, Peter H., “A Generalized Solution of the Orthogonal Procrustes Problem”, Psychometrika vol. 31, No. 1, Mar. 1966, pp. 1-10. [cited by applicant]
Moon, Gyeongsik et al., “Pose2Pose: 3D Positional Pose-Guided 3D Rotational Pose Prediction for Expressive 3D Human Pose and Mesh Estimation”, arXiv, Apr. 1, 2021, 15 pages. [cited by applicant]
Sengupta, Akash et al., “Probabilistic 3D Human Shape and Pose Estimation from Multiple Unconstrained Images in the Wild”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, 10 pages. [cited by applicant]
Song, Jie et al., “Human Body Model Fitting by Learned Gradient Descent”, European Conference on Computer Vision (ECCV), 2020, 17 pages. [cited by applicant]
Sun, Deqing et al., “PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, 18 pages. [cited by applicant]
Sun, Yu et al., “Monocular, One-stage, Regression of Multiple 3D People”, International Conference on Computer Vision (ICCV), 2021, 10 pages. [cited by applicant]
Sun, Yu et al., “Human Mesh Recovery from Monocular Images via a Skeleton-disentangled Representation”, International Conference on Computer Vision (ICCV), 2019, 10 pages. [cited by applicant]
Tung, Hsiao-Yu Fish et al., “Self-supervised Learning of Motion Capture”, 31st Conference on Neural Information Processing Systems (NeurIPS), 2017, 11 pages. [cited by applicant]
Wan, Ziniu et al., “Encoder-decoder with Multi-level Attention for 3D Human Shape and Pose Estimation”, International Conference on Computer Vision (ICCV), 2021, 10 pages. [cited by applicant]
Xu, Yuanlu et al., “DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare”, International Conference on Computer Vision (ICCV), 2019, 11 pages. [cited by applicant]
Zanfir, Andrei et al., “Weakly Supervised 3D Human Pose and Shape Reconstruction with Normalizing Flows”, European Conference on Computer Vision (ECCV), 2020, 20 pages. [cited by applicant]
Zanfir, Andrei et al., “Neural Descent for Visual 3D Human Pose and Shape”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, 10 pages. [cited by applicant]
Zanfir, Mihai et al., “THUNDR: Transformer-based 3D HUmaN Reconstruction with Markers”, International Conference on Computer Vision (ICCV), 2021, 11 pages. [cited by applicant]
Zeng, Wang et al., “3D Human Mesh Regression with Dense Correspondence”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, 15 pages. [cited by applicant]
Zhang, Han et al., “Self-Attention Generative Adversarial Networks”, International Conference on Machine Learning (ICML), 2018, 10 pages. [cited by applicant]
Zhang, Hongwen et al., “Learning 3D Human Shape and Pose from Dense Body Parts”, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020, 18 pages. [cited by applicant]
Zhang, Hongwen et al., “PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop”, International Conference on Computer Vision (ICCV), 2021, 17 pages. [cited by applicant]
Zhang, Tianshu et al., “Object-Occluded Human Shape and Pose Estimation from a Single Color Image”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 7376-7385. [cited by applicant]
Anonymous ECCV Submission, “Taming Transformers for Monocular 3D Human Shape and Pose Estimation”, European Conference on Computer Vision (ECCV), 2022, Paper ID 1410, 18 pages. [cited by applicant]
Bogo, Federica et al., “Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image”, European Conference on Computer Vision (ECCV), 2016, 18 pages. [cited by applicant]
Choi, Hongsuk et al., “Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose”, European Conference on Computer Vision (ECCV), 2021, 27 pages. [cited by applicant]
Dai, Jian S., “Euler-Rodrigues formula variations, quaternion conjugation and intrinsic connections”, Mechanism and Machine Theory, No. 92, 2015, pp. 144-152. [cited by applicant]
Devlin, Jacob et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019, 1… [cited by applicant]
Dosovitskiy, Alexey et al., “An image is worth 16x16 words: Transformers for image recognition at scale”, International Conference on Learning Representations (ICLR), 2021, 21 pages. [cited by applicant]
Hartley, Richard et al., “Multiple View Geometry in Computer Vision”, 2nd Ed., Cambridge University Press, 2003. [cited by applicant]
Ionescu, Catalin et al., “Human3.6M: Large Scale Datasets and Predictive Methods for 3d Human Sensing in Natural Environments”, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2014, 15 pages. [cited by applicant]
Kanazawa, Angjoo et al., “End-to-end Recovery of Human Shape and Pose”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, 10 pages. [cited by applicant]
Kocabas, Muhammed et al., “VIBE: Video Inference for Human Body Pose and Shape Estimation”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, 12 pages. [cited by applicant]
Kolotouros, Nikos et al., “Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop”, International Conference on Computer Vision (ICCV), 2019, 10 pages. [cited by applicant]
Kolotouros, Nikos et al., “Convolutional Mesh Regression for Single-Image Human Shape Reconstruction”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, 10 pages. [cited by applicant]
Kwon, Youngjoong et al., “Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering”, 35th Conference on Neural Information Processing Systems (NIPS), 2021, 15 pages. [cited by applicant]
Li, Ruilong et al., “AI Choreographer: Music Conditioned 3D Dance Generation with AIST++”, International Conference on Computer Vision (ICCV), 2021, 15 pages. [cited by applicant]
Lin, Kevin et al., “End-to-End Human Pose and Mesh Reconstruction with Transformers”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, 17 pages. [cited by applicant]
Lin, Kevin et al., “Mesh Graphormer”, International Conference on Computer Vision (ICCV), 2021, 15 pages. [cited by applicant]
Loper, Matthew et al., “SMPL: A Skinned Multi-Person Linear Model”, ACM Transactions on Graphics, vol. 34, No. 6, Article 248, Nov. 2015, 16 pages. [cited by applicant]
Von Marcard, Timo et al., “Recovering Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera”, European Conference on Computer Vision (ECCV), 2018, 17 pages. [cited by applicant]
Moon, Gyeongsik et al., “I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image”, European Conference on Computer Vision (ECCV), 2020, 23 pages. [cited by applicant]
Peng, Sida et al., “Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 905… [cited by applicant]
Rajasegaran, Jathushan et al., “Tracking People with 3d Representations”, 35th Conference on Neural Information Processing Systems (NIPS), 2021, 11 pages. [cited by applicant]
Sanyal, Soubhik et al., “Learning Realistic Human Reposing using Cyclic Self-Supervision with 3D Shape, Pose, and Appearance Consistency”, International Conference on Computer Vision (ICCV), 2021, 11 pages. [cited by applicant]
Sun, Ke et al., “Deep High-Resolution Representation Learning for Human Pose Estimation”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 5693-5703. [cited by applicant]
Vaswani, Ashish et al., “Attention Is All You Need”, 31st Conference on Neural Information Processing Systems (NIPS), 2017, 11 pages. [cited by applicant]
Wang, Jian et al., “Re-Identification Supervised Texture Generation”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, 11 pages. [cited by applicant]
Xu, Xiangyu et al., “3D Human Pose, Shape and Texture from Low-Resolution Images and Videos”, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021, 15 pages. [cited by applicant]
Xu, Xiangyu et al., “3D Human Texture Estimation from a Single Image with Transformers”, International Conference on Computer Vision (ICCV), 2021, pp. 13849-13858. [cited by applicant]
Carnegie Mellon University; “CMU graphics lab motion capture database”; 2010; retrieved on http://mocap.cs.cmu.edu/. [cited by applicant]
Akhter, Ijaz et al., “Pose-Conditioned Joint Angle Limits for 3D Human Pose Reconstruction”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, 10 pages. [cited by applicant]
Andriluka, Mykhaylo et al., “2D Human Pose Estimation: New Benchmark and State of the Art Analysis”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, 8 pages. [cited by applicant]
Arnab, Anurag et al., “Exploiting temporal context for 3D human pose estimation in the wild”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, 13 pages. [cited by applicant]
Chen, Hanting et al., “Pre-Trained Image Processing Transformer”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, 15 pages. [cited by applicant]
Carion, Nicolas et al., “End-to-End Object Detection with Transformers”, European Conference on Computer Vision (ECCV), 2020, 17 pages. [cited by applicant]
Choi, Hongsuk et al., “Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, 12 pages. [cited by applicant]
Doersch, Carl et al., “Sim2real transfer learning for 3d human pose estimation: motion to the rescue”, 33rd Conference on Neural Information Processing Systems (NIPS), 2019, 14 pages. [cited by applicant]
Dwivedi, Sai Kumar et al., “Learning to Regress Bodies from Images using Differentiable Semantic Rendering”, International Conference on Computer Vision (ICCV), 2022, 15 pages. [cited by applicant]
Georgakis, Georgios et al., “Hierarchical Kinematic Human Mesh Recovery”, European Conference on Computer Vision (ECCV), 2020, 17 pages. [cited by applicant]
Güler, Riza Alp et al., “HoloPose: Holistic 3D Human Reconstruction In-The-Wild”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 10884-10894. [cited by applicant]
Jiang, Wen et al., “Coherent Reconstruction of Multiple Humans from a Single Image”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 82-91. [cited by applicant]
Jiang, Yifan et al., “TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up”, 35th Conference on Neural Information Processing Systems (NIPS), 2021, 19 pages. [cited by applicant]
Joo, Hanbyul et al., “Exemplar Fine-Tuning for 3D Human Model Fitting Towards In-the-Wild 3D Human Pose Estimation”, Facebook AI Research, Oct. 22, 2021, 21 pages. [cited by applicant]
Kanazawa, Angjoo et al., “Learning 3D Human Dynamics from Video”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 5614-5623. [cited by applicant]
Kingma, Diederik P. et al., “ADAM: A method for stochastic optimization”, International Conference on Learning Representations (ICLR), 2017, 15 pages. [cited by applicant]
Kocabas, Muhammed et al., “PARE: Part Attention Regressor for 3D Human Body Estimation”, International Conference on Computer Vision (ICCV), 2021, pp. 11127-11137. [cited by applicant]
Kocabas, Muhammed et al., “SPEC: Seeing People in the Wild with an Estimated Camera”, International Conference on Computer Vision (ICCV), 2021, pp. 11035-11045. [cited by applicant]
Kolotouros, Nikos et al., “Probabilistic Modeling for Human Mesh Recovery”, International Conference on Computer Vision (ICCV), 2021, 6 pages. [cited by applicant]
Lassner, Christoph et al., “Unite the People: Closing the Loop Between 3D and 2D Human Representations”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 6050-6059. [cited by applicant]
Lee, Gun-Hee et al., “Uncertainty-Aware Human Mesh Recovery from Video by Learning Part-Based 3D Dynamics”, International Conference on Computer Vision (ICCV), 2021, pp. 12375-12384. [cited by applicant]
Li, Jiefeng et al., “HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022, 14 pages. [cited by applicant]
Lin, Tsung-Yi et al., “Microsoft COCO: Common Objects in Context”, European Conference on Computer Vision (ECCV), 2015, 15 pages. [cited by applicant]
Examination Report issued for the corresponding Singapore patent application No. 10202202251U, dated Apr. 24, 2022, 5 pages (for informational purposes only). [cited by applicant]
Search Report issued for the corresponding Singapore patent application No. 10202202251U, dated Apr. 24, 2022, 2 pages (for informational purposes only). [cited by applicant]