IP Library › Granted Patent US 12,524,893
Granted Patent B2
US 12,524,893 · App. 18/369,958 · Granted Jan 13, 2026

Shape space generation via progressive correspondence estimation

Inventors: Sanjeev Muralikrishnan (London, GB); Chun-Hao Huang (London, GB); Duygu Ceylan Aksit (London, GB); Niloy J. Mitra (Potters Bar, GB)
Assignee: Adobe Inc.
G06T7/38G06T7/75G06T17/10G06T2207/20081
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,524,893
App. No.
18/369,958
Granted
Jan 13, 2026
Kind
B2
Abstract

In some examples, a computing system access a set of registered three-dimensional (3D) digital shapes. The set of registered 3D digital shapes are registered to a shape template. The computing system determines a linear model for an estimate of the shape space using a first subset of the set of registered 3D digital shapes. The computing system then determines a nonlinear deformation model for the shape space using a second subset of the set of registered 3D digital shapes. An unregistered shape can be registered to the shape space using the linear model and the nonlinear deformation model. The registration can be added to the set of registered 3D digital shapes to update the estimate of the shape space if a shape distance between the registration and the unregistered shape is below a threshold value.

Claims (66)

1 . A method performed by one or more processing devices, comprising:

accessing a set of registered three-dimensional (3D) digital shapes, wherein the set of registered 3D digital shapes are registered to a shape template;

determining a linear model for an estimate of a shape space using a first subset of the set of registered 3D digital shapes;

determining a nonlinear deformation model for the shape space using a second subset of the set of registered 3D digital shapes;

projecting an unregistered shape to the shape space by using the linear model to create an initial registration for the unregistered shape;

predicting an updated registration based on the initial registration using the nonlinear deformation model; and

adding the updated registration to the set of registered 3D digital shapes based on a shape distance between the updated registration and the unregistered shape being below a threshold value for updating the estimate of the shape space to create an updated estimate of the shape space, wherein the updated estimate of the shape space is usable for editing a 3D digital shape in the shape space.

2 . The method of claim 1 , wherein the shape template is a skinned multi-person linear model (SMPL)-based template, wherein the linear model is a principal component analysis (PCA)-based model, comprising multiple shape eigenvectors, and wherein the nonlinear deformation model is a Neural Jacobian Fields (NJF)-based model.

3 . The method of claim 1 , wherein an initial state of the nonlinear deformation model is determined based on the first subset of the set of registered 3D digital shapes.

4 . The method of claim 1 , wherein projecting the unregistered shape to the shape space by using the linear model to create an initial registration for the unregistered shape further comprises:

determining a plurality of optimized pose parameters and a plurality of optimized shape coefficients for the unregistered shape; and

creating the initial registration for the unregistered shape based on the plurality of optimized shape coefficients and the linear model.

5 . The method of claim 4 , wherein the updated registration is posed to match the unregistered shape based on the plurality of optimized pose parameters by using the nonlinear deformation model.

6 . The method of claim 1 , wherein the shape distance is a Chamfer Distance.

7 . The method of claim 1 , further comprising:

adding the updated registration to the first subset of the set of registered 3D digital shapes to create an updated first subset of the set of registered 3D digital shapes;

determining an updated linear model for the shape space using the updated first subset of the set of registered 3D digital shapes;

updating the nonlinear deformation model for the shape space using the second subset of the set of registered 3D digital shapes to create an updated nonlinear deformation model;

projecting a second unregistered shape to the shape space by using the updated linear model to create a second initial registration;

predicting a second updated registration based on the second initial registration using the updated nonlinear deformation model;

determining a second shape distance between the second updated registration and the second unregistered shape; and

adding the second updated registration to the updated first subset of the set of registered 3D digital shapes based on a second shape distance between the second updated registration and the second unregistered shape being below the threshold value to create a further updated first set of registered 3D digital shapes and a further updated estimate of the shape space.

8 . A system, comprising:

a memory component;

a processing device coupled to the memory component, the processing device to perform operations comprising:

accessing a set of registered three-dimensional (3D) digital shapes, wherein the set of registered 3D digital shapes are registered to a shape template;

determining a linear model for an estimate of a shape space using a first subset of the set of registered 3D digital shapes;

determining a nonlinear deformation model for the shape space using a second subset of the set of registered 3D digital shapes;

projecting an unregistered shape to the shape space by using the linear model to create an initial registration for the unregistered shape;

predicting an updated registration based on the initial registration using the nonlinear deformation model; and

adding the updated registration to the first subset of the set of registered 3D digital shapes based on a shape distance between the updated registration and the unregistered shape being below a threshold value to create an updated first subset of the set of registered 3D digital shapes for updating the estimate of the shape space to create an updated estimate of the shape space, wherein the updated estimate of the shape space is usable for editing a 3D digital shape in the shape space.

9 . The system of claim 8 , wherein the shape template is a skinned multi-person linear model (SMPL)-based template, wherein the linear model is a principal component analysis (PCA)-based model, comprising multiple shape eigenvectors, and wherein the nonlinear deformation model is a Neural Jacobian Fields (NJF)-based model.

10 . The system of claim 8 , wherein an initial state of the nonlinear deformation model is determined based on the first subset of the set of registered 3D digital shapes.

11 . The system of claim 8 , wherein projecting the unregistered shape to the shape space by using the linear model further comprises:

determining a plurality of optimized pose parameters and a plurality of optimized shape coefficients for the unregistered shape; and

identify a registered shape from the first subset of the set of registered 3D digital shapes best matching the unregistered shape based on the plurality of optimized shape coefficients.

12 . The system of claim 11 , wherein the updated registration is posed to match the unregistered shape based on the plurality of optimized pose parameters by using the nonlinear deformation model.

13 . The system of claim 8 , wherein the shape distance is a Chamfer Distance.

14 . The system of claim 8 , wherein the processing device is to perform further operations comprising:

determining an updated linear model for the shape space using the updated first subset of the set of registered 3D digital shapes;

updating the nonlinear deformation model for the shape space using the second subset of the set of registered 3D digital shapes to create an updated nonlinear deformation model;

projecting a second unregistered shape to the shape space by using the updated linear model to create a second initial registration;

predicting a second updated registration based on the second initial registration using the updated nonlinear deformation model;

determining a second shape distance between the second updated registration and the second unregistered shape; and

adding the second updated registration to the updated first subset of the set of registered 3D digital shapes based on a second shape distance between the second updated registration and the second unregistered shape being below the threshold value to create a further updated first set of registered 3D digital shapes and a further updated estimate of the shape space.

15 . A non-transitory computer-readable medium, storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:

accessing a set of registered three-dimensional (3D) digital shapes, wherein the set of registered 3D digital shapes are registered to a shape template;

a step for determining a linear model for an estimate of a shape space using a first subset of the set of registered 3D digital shapes;

a step for obtaining a nonlinear deformation model for the shape space using a second subset of the set of registered 3D digital shapes;

projecting an unregistered shape to the shape space by using the linear model to create an initial registration for the unregistered shape;

predicting an updated registration based on the initial registration using the nonlinear deformation model; and

adding the updated registration to the set of registered 3D digital shapes based on a shape distance between the updated registration and the unregistered shape being below a threshold value for updating the estimate of the shape space to create an updated estimate of the shape space, wherein the updated estimate of the shape space is usable for editing a 3D digital shape.

16 . The non-transitory computer-readable medium of claim 15 , wherein the shape template is a skinned multi-person linear model (SMPL)-based template, wherein the linear model is a principal component analysis (PCA)-based model, comprising multiple shape eigenvectors, and wherein the nonlinear deformation model is a Neural Jacobian Fields (NJF)-based model.

17 . The non-transitory computer-readable medium of claim 15 , wherein an initial state of the nonlinear deformation model is determined based on the first subset of the set of registered 3D digital shapes.

18 . The non-transitory computer-readable medium of claim 15 , wherein projecting the unregistered shape to the shape space by using the linear model further comprises:

determining a plurality of optimized pose parameters and a plurality of optimized shape coefficients for the unregistered shape; and

identify a registered shape from the first subset of the set of registered 3D digital shapes best matching the unregistered shape based on the plurality of optimized shape coefficients.

19 . The non-transitory computer-readable medium of claim 15 , wherein the updated registration is posed to match the unregistered shape based on the plurality of optimized pose parameters by using the nonlinear deformation model.

20 . The non-transitory computer-readable medium of claim 15 , wherein the executable instructions, which when executed by a processing device, cause the processing device to perform further operations comprising:

adding the updated registration to the first subset of the set of registered 3D digital shapes to create an updated first subset of the set of registered 3D digital shapes;

determining an updated linear model for the shape space using the updated first subset of the set of registered 3D digital shapes;

updating the nonlinear deformation model for the shape space using the second subset of the set of registered 3D digital shapes to create an updated nonlinear deformation model;

projecting a second unregistered shape to the shape space by using the updated linear model to create a second initial registration;

predicting a second updated registration based on the second initial registration using the updated nonlinear deformation model;

determining a second shape distance between the second updated registration and the second unregistered shape; and

adding the second updated registration to the updated first subset of the set of registered 3D digital shapes based on a second shape distance between the second updated registration and the second unregistered shape being below the threshold value to create a further updated first set of registered 3D digital shapes and a further updated estimate of the shape space.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 19, 2023
From: MURALIKRISHNAN, SANJEEV; HUANG, CHUN-HAO; AKSIT, DUYGU CEYLAN; MITRA, NILOY J.
To: ADOBE INC.
Reel/Frame 064949/0297 →
Continuity (1)
Related Publication 20250095172A1 · Mar 20, 2025
References Cited (62)
US 10813715B1 · Chojnowski · 2020 [cited by examiner]
US 12236517B2 · Bradley · 2025 [cited by examiner]
US 20160371542A1 · Sugita · 2016 [cited by examiner]
US 20180315230A1 · Black · 2018 [cited by examiner]
US 20200058137A1 · Pujades · 2020 [cited by examiner]
US 20230169727A1 · Sminchisescu · 2023 [cited by examiner]
US 20230281921A1 · Miao · 2023 [cited by examiner]
SizeUSA Dataset, Available online at: https://www.tc2.com/size-USA.html, 2017. [cited by applicant]
Aigerman et al., Neural Jacobian Fields: Learning Intrinsic Mappings of Arbitrary Meshes, ACM Trans. Graph., vol. 41, No. 4, Jul. 2022, pp. 1-17. [cited by applicant]
Anguelov et al., SCAPE: Shape Completion and Animation of People, ACM Transactions on Graphics (TOG), ACM, 2005, pp. 408-416. [cited by applicant]
Aubry et al., The Wave Kernel Signature: A Quantum Mechanical Approach to Shape Analysis, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Nov. 6-13, 2011, 8 pages. [cited by applicant]
Azencot et al., Consistent Shape Matching via Coupled Optimization, Computer Graphics Forum, vol. 38, No. 5, Available Online at: https://doi.org/10.1111/cgf.13786, Aug. 12, 2019, 13 pages. [cited by applicant]
Bellekens et al., A Survey of Rigid 3D Pointcloud Registration Algorithms, Conference: Ambient 2014: The Fourth International Conference on Ambient Computing, Applications, Services and Technologies, 2014, pp. 8-13. [cited by applicant]
Besl et al., A Method for Registration of 3-D Shapes, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, No. 2, Feb. 1992, pp. 239-256. [cited by applicant]
Blanz et al., A Morphable Model for the Synthesis of 3D Faces, SIGGRAPH '99, Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, Jul. 1999, pp. 187-194. [cited by applicant]
Bogo et al., Detailed Full-Body Reconstructions of Moving People from Monocular RGB-D Sequences, In ICCV '11 Proceedings International Conference on Computer Vision, Dec. 2015, pp. 2300-2308. [cited by applicant]
Boscaini et al., Learning Shape Correspondence with Anisotropic Convolutional Neural Networks, Computer Vision and Pattern Recognition, May 20, 2016, 9 pages. [cited by applicant]
Chen et al., Object Modeling by Registration of Multiple Range Images, Proceedings of the 1991 Institute of Electrical and Electronics Engineers, International Conference on Robotics and Automation, Sacramento, Californ… [cited by applicant]
Chen et al., Robust Nonrigid Registration by Convex Optimization, 2015 IEEE International Conference on Computer Vision (ICCV), Dec. 7-13, 2015, 10 pages. [cited by applicant]
Cootes et al., Active Shape Models-their Training and Application, Computer Vision and Image Understanding, vol. 61, No. 1, Jan. 1995, pp. 38-59. [cited by applicant]
Dale et al., Video face replacement., In ACM Transactions on Graphics (Proc. SIGGRAPH Asia), Dec. 2011, pp. 1-10. [cited by applicant]
Deng et al., A Survey of Non-rigid 3D Registration, STAR—State of The Art Report, vol. 41, No. 2, 2022, 31 pages. [cited by applicant]
Deng et al., NASA: Neural Articulated Shape Approximation, Available Online at: https://arxiv.org/pdf/1912.03207.pdf, Aug. 2020, 21 pages. [cited by applicant]
Dou et al., 3D Scanning Deformable Objects with a Single RGBD Sensor, Available Online at: https://www.cs.toronto.edu/˜jtaylor/papers/CVPR2015-ScanningDeformableObjects.pdf, 2015, pp. 493-501. [cited by applicant]
Dou et al., Fusion4D: Real-time Performance Capture of Challenging Scenes, ACM Transactions on Graphics, vol. 35, Issue 4, Jul. 11, 2016, pp. 1-13. [cited by applicant]
Egger et al., 3D Morphable Face Models—Past, Present and Future, Computer Vision and Pattern Recognition, Apr. 16, 2020, 39 pages. [cited by applicant]
Garrido et al., Automatic Face Reenactment, Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Sep. 2014, 8 pages. [cited by applicant]
Ghorbani et al., MoVi: a Large Multi-purpose Human Motion and Video Dataset, PloS One, vol. 16, No. 6, Jun. 17, 2021, pp. 1-15. [cited by applicant]
Groueix et al., 3D-Coded: 3D Correspondences by Deep Deformation, Proceedings of the European Conference on Computer Vision (ECCV), Jul. 27, 2018, pp. 1-23. [cited by applicant]
Hirshberg et al., Coregistration: Simultaneous Alignment and Modeling of Articulated 3D Shape, Conference: Proceedings of the 12th European Conference on Computer Vision—vol. Part VI, Oct. 2012, 14 pages. [cited by applicant]
Hu et al., Avatar Digitization from a Single Image for Real-time Rendering, ACM Transactions on Graphics, vol. 36, No. 6, Nov. 20, 2017, pp. 195:1-195:14. [cited by applicant]
Huang et al., A Bayesian Approach to Multi-view 4D Modeling, International Journal of Computer Vision, vol. 116, No. 2, Jun. 2015, 21 pages. [cited by applicant]
Huang et al., ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators, Computer Vision and Pattern Recognition, Sep. 21, 2021, 17 pages. [cited by applicant]
Huang et al., Tracking-by-Detection of 3D Human Shapes: From Surfaces to Volumes, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, No. 8, Aug. 2018, 15 pages. [cited by applicant]
Joo et al., Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies, IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 18, 2018, pp. 8320-8329. [cited by applicant]
Li et al., Global Correspondence Optimization for Non-Rigid Registration of Depth Scans, Computer Graphics Forum, vol. 27, No. 5, Available Online at: http://dl.acm.org/citation.cfm?id=1731309.1731326, Jul. 2, 2008, pp.… [cited by applicant]
Loper et al., SMPL: A Skinned Multi-Person Linear Model, ACM Transactions on Graphics, vol. 34, No. 6, Nov. 2, 2015, pp. 1-16. [cited by applicant]
Mahmood et al., AMASS: Archive of Motion Capture as Surface Shapes, Computer Vision and Pattern Recognition, Apr. 5, 2019, 12 pages. [cited by applicant]
Mihajlovic et al., COAP: Compositional Articulated Occupancy of People, Computer Vision and Pattern Recognition, Apr. 13, 2022, 12 pages. [cited by applicant]
Mihajlovic et al., LEAP: Learning Articulated Occupancy of People, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Apr. 14, 2021, 17 pages. [cited by applicant]
Monti et al., Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs, Computer Vision and Pattern Recognition, Available Online at: https://arxiv.org/pdf/1611.08402.pdf, Dec. 6, 2016, pp. 1-13. [cited by applicant]
Muralikrishnan et al., GLASS: Geometric Latent Augmentation for Shape Spaces, Computer Vision and Pattern Recognition, Apr. 29, 2022, 14 pages. [cited by applicant]
Osman et al., STAR: Sparse Trained Articulated Human Body Regressor, Computer Vision and Pattern Recognition, Aug. 19, 2020, 18 pages. [cited by applicant]
Osman et al., SUPR: A Sparse Unified Part-Based Human Representation, Computer Vision and Pattern Recognition, Oct. 25, 2022, 43 pages. [cited by applicant]
Pavlakos et al., Expressive Body Capture: 3D Hands, Face, and Body from a Single Image, In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Apr. 11, 2019, pp. 10975-10985. [cited by applicant]
Pons-Moll et al., Dyna: A Model of Dynamic Human Shape in Motion, ACM Transactions on Graphics, vol. 34, No. 4, Jul. 27, 2015, pp. 1-14. [cited by applicant]
Qi et al., PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Apr. 10, 2017, pp. 652-660. [cited by applicant]
Ren et al., Discrete Optimization for Shape Matching, Computer Graphics Forum, vol. 40, No. 5, Aug. 2021, 16 pages. [cited by applicant]
Robinette et al., Civilian American and European Surface Anthropometry Resource (CAESAR) Final Report, Technical Report AFRL-HE-WP-TR-2002-0169, US Air Force Research Laboratory, Jun. 2002, 70 pages. [cited by applicant]
Romero et al., Embodied Hands: Modeling and Capturing Hands and Bodies Together, ACM Transactions on Graphics, vol. 36, No. 6, Nov. 2017, pp. 1-17. [cited by applicant]
Salti et al., SHOT: Unique Signatures of Histograms for Surface and Texture Description, Computer Vision and Image Understanding, vol. 125, Aug. 2014, pp. 251-264, Abstract pp. 1-8. [cited by applicant]
Sorkine et al., As-Rigid-As-Possible Surface Modeling, SGP '07: Proceedings of the fifth Eurographics symposium on Geometry processing, vol. 4, Jul. 2007, 8 pages. [cited by applicant]
Sorkine et al., Laplacian Surface Editing, in Proceedings of the Eurographics/ACM SIGGRAPH Symposium on Geometry Processing, Jul. 2004, pp. 175-184. [cited by applicant]
Sun et al., A Concise and Provably Informative Multi-Scale Signature Based on Heat Diffusion, Computer Graphics Forum, vol. 28, No. 5, Aug. 31, 2009, 10 pages. [cited by applicant]
Thies et al., Real-Time Expression Transfer for Facial Reenactment, Association for Computing Machinery Transactions on Graphics, vol. 34, No. 6, Oct. 2015, 14 pages. [cited by applicant]
Tian et al., Recovering 3D Human Mesh from Monocular Images: A Survey, Available Online at: https://arxiv.org/pdf/2203.01923.pdf, Aug. 25, 2023, pp. 1-25. [cited by applicant]
Tiwari et al., Neural-GIF: Neural Generalized Implicit Functions for Animating People in Clothing, IEEE/CVF International Conference on Computer Vision (ICCV), Aug. 2021, pp. 11708-11718. [cited by applicant]
Wei et al., Dense Human Body Correspondences Using Convolutional Networks, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, Jun. 2016, pp. 1544-1553. [cited by applicant]
Wood et al., Fake it Till You Make it: Face Analysis in the Wild Using Synthetic Data Alone, Available Online at: https://arxiv.org/pdf/2109.15102.pdf, Oct. 5, 2021, 11 pages. [cited by applicant]
Xie et al., Neural Fields in Visual Computing and Beyond, Eurographics, vol. 41, No. 2, Available Online at: https://arxiv.org/pdf/2111.11426.pdf, May 24, 2022, 36 pages. [cited by applicant]
Xu et al., DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare, IEEE/CVF International Conference on Computer Vision (ICCV), Oct. 2019, pp. 7760-7770. [cited by applicant]
Xu et al., GHUM & GHUML: Generative 3D Human Shape and Articulated Pose Models, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 1, 2020, pp. 6184-6193. [cited by applicant]