IP Library › Granted Patent US 12,430,851
Granted Patent B2
US 12,430,851 · App. 18/662,020 · Granted Sep 30, 2025

Synthesizing high resolution 3D shapes from lower resolution representations for synthetic data generation systems and applications

Inventors: Tianchang Shen (Markham, CA); Jun Gao (Toronto, CA); Kangxue Yin (Toronto, CA); Ming-Yu Liu (San Jose, CA); Sanja Fidler (Toronto, CA)
Assignee: NVIDIA Corporation
G06T17/20G06T7/50G06T2207/10028G06T2207/20081G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,430,851
App. No.
18/662,020
Granted
Sep 30, 2025
Kind
B2
Abstract

In various examples, a deep three-dimensional (3D) conditional generative model is implemented that can synthesize high resolution 3D shapes using simple guides—such as coarse voxels, point clouds, etc.—by marrying implicit and explicit 3D representations into a hybrid 3D representation. The present approach may directly optimize for the reconstructed surface, allowing for the synthesis of finer geometric details with fewer artifacts. The systems and methods described herein may use a deformable tetrahedral grid that encodes a discretized signed distance function (SDF) and a differentiable marching tetrahedral layer that converts the implicit SDF representation to an explicit surface mesh representation. This combination allows joint optimization of the surface geometry and topology as well as generation of the hierarchy of subdivisions using reconstruction and adversarial losses defined explicitly on the surface mesh.

Claims (88)

1. A method comprising:

computing a signed distance field (SDF) corresponding to an initial polyhedral grid, the initial polyhedral grid being representative of an object in a scene;

identifying one or more first vertices and one or more first edges associated with the initial polyhedral grid;

generating an updated polyhedral grid by at least:

computing one or more position offsets for the one or more first vertices; and

adding, based at least on the one or more position offsets, one or more second vertices and one or more second edges;

computing an updated SDF using the SDF and the updated polyhedral grid; and

generating a polygon mesh of the object based at least on the updated SDF.

2. The method of claim 1 , further comprising:

identifying one or more surface volumes of the initial polyhedral grid that correspond to a surface of the object; and

generating a graph corresponding to the one or more first vertices and the one or more first edges of the one or more surface volumes,

wherein the generating of the updated polyhedral grid is based at least on the graph.

3. The method of claim 1 , wherein:

the SDF includes one or more first SDF values associated with the one or more first vertices; and

the computing the updated SDF comprises computing one or more second SDF values by performing interpolation on the one or more first SDF values based at least on the one or more position offsets.

4. The method of claim 1 , wherein the generating the updated polyhedral grid is based at least on applying a hierarchy of subdivisions to one or more surfaces of the initial polyhedral grid.

5. The method of claim 1 , wherein:

the initial polyhedral grid is associated with a first resolution; and

the updated polyhedral grid is associated with a second resolution that is greater than the first resolution.

6. The method of claim 1 , wherein the computing the SDF is at least by:

computing one or more feature vectors; and

computing, using one or more neural networks and based at least on the one or more feature vectors, one or more SDF values for the one or more first vertices of the initial polyhedral grid, the SDF including the one or more SDF values.

7. The method of claim 1 , further comprising generating, based at least on subdividing the polygon mesh, a surface representation associated with the object.

8. A system comprising:

one or more processors to:

generate an initial polyhedral grid representative of an object in a scene, the initial polyhedral grid including first vertices and one or more first edges;

determine that a first portion of the first vertices are associated with one or more positive values and a second portion of the first vertices are associated with one or more negative values;

generate, based at least on the first portion of the first vertices being associated with the one or more positive values and the second portion of the first vertices being associated with the one or more negative values, an updated polyhedral grid by at least adding one or more second vertices and one or more second edges to the initial polyhedral grid; and

generate a polygon mesh of the object based at least on the updated polyhedral grid.

9. The system of claim 8 , wherein the one or more processors are further to:

determine one or more position offsets associated with the first vertices,

wherein the generation of the updated polyhedral grid is further based at least on the one or more position offsets.

10. The system of claim 8 , wherein the one or more processors are further to:

identify one or more surface volumes of the initial polyhedral grid that correspond to a surface of the object; and

generate a graph corresponding to the first vertices and the one or more first edges of the one or more surface volumes,

wherein the generation of the updated polyhedral grid is further based at least on the graph.

11. The system of claim 8 , wherein the one or more processors are further to:

determine one or more first signed distance field (SDF) values associated with the first vertices; and

determine, based at least on the one or more first SDF values, one or more second SDF values associated with the one or more second vertices,

wherein the generation of the polygon mesh is further based at least on the one or more second SDF values.

12. The system of claim 11 , wherein the determination of the one more second SDF values associated with the one or more second vertices is based at least on performing interpolation associated with the one or more first SDF values.

13. The system of claim 8 , wherein:

the initial polyhedral grid is associated with a first resolution; and

the updated polyhedral grid is associated with a second resolution that is greater than the first resolution.

14. The system of claim 8 , wherein the one or more processors are further to generate, based at least on subdividing the polygon mesh, a surface representation associated with the object.

15. The system of claim 8 , wherein the system is comprised in at least one of:

a control system for an autonomous or semi-autonomous machine;

a perception system for an autonomous or semi-autonomous machine;

a system for performing simulation operations;

a system for performing light transport simulation;

a system for performing collaborative content creation for 3D assets;

a system for performing deep learning operations;

a system implemented using an edge device;

a system implemented using a robot;

a system for performing conversational AI operations;

a system for generating synthetic data;

a system incorporating one or more virtual machines (VMs);

a system implemented at least partially in a data center; or

a system implemented at least partially using cloud computing resources.

16. One or more processors comprising:

processing circuitry to:

determine one or more first vertices and one or more first edges associated with a polyhedral grid representative of an object;

determine one or more offsets using at least the one or more first vertices;

generate an updated polyhedral grid by at least adding one or more second vertices and one or more second edges to the one or more first vertices and the one or more first edges based at least on the one or more offsets; and

generate a surface representation associated with the object based at least on the updated polyhedral grid.

17. The one or more processors of claim 16 , wherein the processing circuitry is further to:

determine that at least a first portion of the one or more first vertices are associated with one or more positive values and a second portion of the one or more first vertices are associated with one or more negative values,

wherein the generation of the updated polyhedral grid is further based at least on the first portion of the one or more first vertices being associated with one or more positive values and the second portion of the one or more first vertices being associated with one or more negative values.

18. The one or more processors of claim 16 , wherein the one or more processors are comprised in at least one of:

a control system for an autonomous or semi-autonomous machine;

a perception system for an autonomous or semi-autonomous machine;

a system for performing simulation operations;

a system for performing light transport simulation;

a system for performing collaborative content creation for 3D assets;

a system for performing deep learning operations;

a system implemented using an edge device;

a system implemented using a robot;

a system for performing conversational AI operations;

a system for generating synthetic data;

a system incorporating one or more virtual machines (VMs);

a system implemented at least partially in a data center; or

a system implemented at least partially using cloud computing resources.

19. The one or more processors of claim 16 , wherein the processing circuitry is further to:

determine a signed distance field (SDF) based at least on the updated polyhedral grid,

wherein the surface representation is generated based at least on the SDF.

20. The one or more processors of claim 16 , wherein the processing circuitry is further to:

generate a graph corresponding to the one or more first vertices and the one or more first edges,

wherein the updated polyhedral grid is further generated based at least on the graph.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 11, 2024
From: SHEN, TIANCHANG; GAO, JUN; YIN, KANGXUE; LIU, MING-YU; FIDLER, SANJA
To: NVIDIA CORPORATION
Reel/Frame 067685/0524 →
Continuity (3)
Continuation 17718172 · Apr 11, 2022
Provisional Application 63194693 · May 28, 2021
Related Publication 20240296627A1 · Sep 5, 2024
References Cited (73)
US 11983815B2 · Shen et al. · 2024 [cited by applicant]
US 20090244065A1 · Storti et al. · 2009 [cited by applicant]
US 20160364907A1 · Schoenberg · 2016 [cited by applicant]
US 20180031721A1 · Etiene Queiroz · 2018 [cited by examiner]
US 20200387739A1 · Williams et al. · 2020 [cited by applicant]
US 20210149022A1 · Kehl et al. · 2021 [cited by applicant]
US 20210272345A1 · Lesser et al. · 2021 [cited by applicant]
US 20220392162A1 · Shen et al. · 2022 [cited by applicant]
WO 2022250796A1 · 2022 [cited by applicant]
Wu, et al.; “Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling,” Conference on Neural Information Processing Systems, 2016, 9 pgs. [cited by applicant]
Wang, et al.; “O-CNN: Octree-Based Convolutional Neural Networks for 3D Shape Analysis,” ACM Transactions on Graphics (SIGGRAPH), vol. 36, Issue 4, Article 72, 2017, 11 pgs. [cited by applicant]
Wang, et al.; “Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images,” In Proceedings of the European Conference on Computer Vision (ECCV), 2018, 16 pgs. [cited by applicant]
Wang, et al.; “DISN: Deep Implicit Surface Network for High-Quality Single-View 3D Reconstruction,” Conference on Neural Information Processing Systems, 2019, 11 pgs. [cited by applicant]
Wang, et al.; “Deep Octree-Based CNNs with Output-Guided Skip Connections for 3D Shape and Scene Completion,” In Proceedings of the IEEE/CVF Conf on Comp Vis and Pattern Recog Workshops, 2020, 8 pgs. [cited by applicant]
Williams, et al.; “Neural Splines: Fitting 3D Surfaces with Infinity-Wide Neural Networks,” arXiv.2006.13782v2, 2020, 24 pgs. [cited by applicant]
Yin, et al.; “COALESCE: Component Assembly by Learning to Synthesize Connections,” arXiv:2008.01936v2, 2020, 20 pgs. [cited by applicant]
Zhu, et al.; “SCORES: Shape Composition with Recursive Substructure Priors,” ACM Transactions on Graphics, vol. 37, No. 6, Nov. 2018, 14 pgs. [cited by applicant]
Wenzheng Chen et al: “Learning to Predict 3D objects with an Interpolation-based Differentiable Renderer”, ARXIV.org; Cornell University Library, Aug. 3, 2019, pp. 1-11. [cited by applicant]
D'Otreppe Vinciane et al: “Generating Smooth Surface Meshes From Multi-Region Medical Images”, International Journal for Numerical Methods in Biomedical Engineering, vol. 28, No. 6-7, Oct. 17, 2011, pp. 642-660. [cited by applicant]
Klacansky Pavol et al: “Fast and Exact Fiber Surfaces for Tetrahedral Meshes”, IEEE Transaction On Visualization and Computer Graphics, IEEE, USA, vol. 23, No. 7, Jul. 1, 2017, pp. 1782-1795. [cited by applicant]
Liu Hsueh-Ti Derek Hsuehtil@CS Toronto Edu et al: “Neural Subdivision” ACM Transaction On Graphics, ACM, NY, US, vol. 39, No. 4, Jul. 8, 2020, p. 124: 1-124: 16. [cited by applicant]
Edoardo Remelli et al: “MeshSDF: Differentiable Iso-Surface Extraction” Arxiv.Org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Oct. 31, 2020. [cited by applicant]
Tianchang Shen et al: “Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis” Arxiv.Org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, Nov. 8, 2021. [cited by applicant]
Shen, Tianchang; International Search Report and Written Opinion for PCT Application No. PCT/US2022/024306, filed Apr. 11, 2022, mailed Aug. 8, 2022, 14 pgs. [cited by applicant]
Park, J J., et al.; “DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation”; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 165-174 (2019). [cited by applicant]
Shen, et al.; Non-Final Office Action for U.S. Appl. No. 17/718,172, filed Apr. 11, 2022, mailed Jul. 6, 2023, 38 pgs. [cited by applicant]
Shen, Tianchang; Final Office Action for U.S. Appl. No. 17/718,172, filed Apr. 11, 2022, mailed Oct. 23, 2023, 19 pgs. [cited by applicant]
Shen, Tianchang; International Preliminary Report on Patentability for PCT Application No. PCT/US2022/024306, filed Apr. 11, 2022, mailed Dec. 7, 2023, 10 pgs. [cited by applicant]
Shen, Tianchang; Notice of Allowance for U.S. Appl. No. 17/718,172, filed Apr. 11, 2022, mailed Jan. 11, 2024, 7 pgs. [cited by applicant]
Mildenhall, et al.; “NeRF: Representing Scenes As Neural Radiance Fields for View Synthesis,” https://arxiv.org/abs/2003.08934, Aug. 3, 2020, 25 pgs. [cited by applicant]
Brock, et al.; “Generative and Discriminative Voxel Modeling with Convolutional Neural Networks,” arXiv:1608.04236v2, pp. 1-9 (Aug. 16, 2016). [cited by applicant]
Chen, et al.; “On Visual Similarity Based 3D Model Retrieval,” Eurographics, vol. 22, No. 3, pp. 223-232 (2003). [cited by applicant]
Chang, et al.; “ShapeNet: An Information-Rich 3D Model Repository,” arXiv:1512.03012v1, Dec. 9, 2015, 11 pgs. [cited by applicant]
Choy, et al.; “3D-R2N2: A Unified Approach for Single and Multi-View 3D Object Reconstruction,” European Conference on Computer Vision, 2016, 17 pgs. [cited by applicant]
Chen, et al.; “BSP-Net: Generating Compact Mashes via Binary Space Partitioning,” arXiv.1911.06971v1, 2019, 10 pgs. [cited by applicant]
Chen, et al.; “Learning Implicit Fields for Generative Shape Modeling,” Conference on Computer Vision and Pattern Recognition, IEEE, 2019, 10 pgs. [cited by applicant]
Chen, et al.; “DECOR-GAN: 3D Shape Detailization by Conditional Refinement,” Conference on Computer Vision and Pattern Recognition, IEEE, 2021, 10 pgs. [cited by applicant]
Doi, et al.; “An Efficient Method of Triangulating Equi-Valued Surfaces by Using Tetrahedral Cells,” IEICE Transactions, vol. E74, No. 1, Jan. 1991, 11 pgs. [cited by applicant]
Doran, et al.; “Isosurface Stuffing Improved: Acute Lattices and Feature Matching,” ACM SIGGRAPH, Jul. 2013, 1 pg. [cited by applicant]
Dai, et al.; “Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis,” Conference on Computer Vision and Pattern Recognition, IEEE, 2017, 10 pgs. [cited by applicant]
Dai, et al.; “ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans,” Conference on Computer Vision and Pattern Recognition, IEEE, 2018, 10 pgs. [cited by applicant]
Duan, et al.; “Curriculum DeepSDF,” In European Conference on Computer Vision, arXiv.2003.08593v1, Mar. 19, 2020, 17 pgs. [cited by applicant]
Davies, et al.; “Overfit Neural Networks as a Compact Shape Representation,” arXiv.2009.09808v1, Sep. 17, 2020, 9 pgs. [cited by applicant]
Deng, et al.; “Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondence,” arXiv.2011.13650v1, Nov. 27, 2020, 15 pgs. [cited by applicant]
Groueix, et al.; “A Papier-Mache Approach to Learning 3D Surface Generation,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, 9 pgs. [cited by applicant]
Gkioxari, et al.; “Mesh R-CNN,” In Proceedings of the IEEE International Conference on Computer Vision, 2019, 11 pgs. [cited by applicant]
Gao, et al.; “SDM-NET: Deep Generative Network for Structured Deformable Mesh,” ACM Transactions on Graphics, vol. 38, No. 6, Article 243, Nov. 2019, 15 pgs. [cited by applicant]
Gao, et al.; “Learning Deformable Tetrahedral Meshes for 3D Reconstruction,” Conference on Neural Information Processing Systems, Nov. 3, 2020, 12 pgs. [cited by applicant]
Hane, et al.; “Hierarchical Surface Prediction for 3D Object Reconstruction,” In International Conference on 3D Vision, IEEE, 2017, 12 pgs. [cited by applicant]
Hanocka, et al.; “Point2Mesh: A Self-Prior for Deformable Meshes,” ACM Trans. Graph, vol. 39, No. 4, Article 126, 2020, 12 pgs. [cited by applicant]
Hao, et al.; “DualSDF: Semantic Shape Manipulation Using A Two-Level Representation,” IEEE, 2020, 11 pgs. [cited by applicant]
Kleineberg, et al.; “Adversarial Generation of Continuous Implicit Shape Representations,” arXiv:2002.00349v2, 2020, 6 pgs. [cited by applicant]
Loop; “Smooth Subdivision Surfaces Based On Triangles,” University of Utah, 1987, 74 pgs. [cited by applicant]
Lorensen, et al.; “Marching Cubes: A High Resolution 3D Surface Construction Algorithm,” ACM SIGGRAPH Computer Graphics, 1987, 7 pgs. [cited by applicant]
Liao, et al.; “Deep Marching Cubes: Learning Explicit Surface Representations,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, 10 pgs. [cited by applicant]
Liu, et al.; “Point-Voxel CNN for Efficient 3D Deep Learning,” Conference on Neural Information Processing Systems, 2019, 11 pgs. [cited by applicant]
Li, et al.; “D2IM-Net: Learning Detail Disentangled Implicit Fields from Single Images,” arXiv.2012.06650v2, 2020, 16 pgs. [cited by applicant]
Maturana, et al.; “VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition,” International Conference on Intelligent Robots and Systems, 2015, 7 pgs. [cited by applicant]
Mao, et al.; “Least Squares Generative Adversarial Networks,” Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017, 9 pgs. [cited by applicant]
Murthy, et al.; “Kaolin: A Pytorch Library for Accelerating 3D Deep Learning Research,” arXiv.1911.05063v2, 2019, 7 pgs. [cited by applicant]
Mescheder, et al.; “Occupancy Networks: Learning 3D Reconstruction in Function Space,” Computer Vision and Pattern Recognition, 2019, 11 pgs. [cited by applicant]
Nash, et al.; “PolyGen: An Autoregressive Generative Model of 3D Meshes,” International Conference on Machine Learning, 2020, 10 pgs. [cited by applicant]
Peng, et al.; “Convolutional Occupancy Networks,” arXiv.2003.04618v2, Aug. 1, 2020, 17 pgs. [cited by applicant]
Paschalidou, et al.; “Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks,” Computer Vision and Pattern Recognition, IEEE, 2021, 12 pgs. [cited by applicant]
Riegler, et al.; “OctNet: Learning Deep 3D Representations at High Resolutions,” Conference on Computer Vision and Pattern Recognition, IEEE, 2017, 10 pgs. [cited by applicant]
Riegler, et al.; “OctNetFusion: Learning Depth Fusion from Data,” arXiv:1704.01047v3, IEEE, Oct. 31, 2017, 10 pgs. [cited by applicant]
Sung, et al.; “ComplementMe: Weakly-Supervised Component Suggestions for 3D Modeling,” ACM Transactions on Graphics, vol. 36, No. 6, Article 226, Nov. 2017, 12 pgs. [cited by applicant]
Saito, et al.; “PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization,” In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, 11 pgs. [cited by applicant]
Saito, et al.; “PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020, 10 pgs. [cited by applicant]
Tatarchenko, et al.; “Octree Generating Networks: Efficient Convolutional Architectures for High-Resolution 3D Outputs,” In IEEE International Conference on Computer Vision (ICCV), 2017, 9 pgs. [cited by applicant]
Tulsiani, et al.; “Learning Shape Abstractions by Assembling Volumetric Primitives,” In Computer Vision and Pattern Recognition (CVPR), 2017, 9 pgs. [cited by applicant]
Takikawa, et al.; “Neural Geometric Level of Detail: Real-Time Rendering with Implicit 3D Shapes,” arXiv:2101.10994v1, 2021, 16 pgs. [cited by applicant]
Wu, et al.; “3D ShapeNets: A Deep Representation for Volumetric Shapes,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 9 pgs. [cited by applicant]