IP Library › Granted Patent US 12,602,890
Granted Patent B2
US 12,602,890 · App. 18/361,587 · Granted Apr 14, 2026

Neural vector fields for 3D shape generation

Inventors: Zekun Hao (New York, NY); Ming-Yu Liu (San Jose, CA); Arun Mohanray Mallya (San Jose, CA)
Assignee: NVIDIA Corporation
G06T19/20G06F30/10G06T2210/56G06T2219/2021
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,602,890
App. No.
18/361,587
Granted
Apr 14, 2026
Kind
B2
Abstract

Synthesis of high-quality 3D shapes with smooth surfaces has various creative and practical use cases, such as 3D content creation and CAD modeling. A vector field decoder neural network is trained to predict a generative vector field (GVF) representation of a 3D shape from a latent representation (latent code or feature volume) of the 3D shape. The GVF representation is agnostic to surface orientation, all dimensions of the vector field vary smoothly, the GVF can represent both watertight and non-watertight 3D shapes, and there is a one-to-one mapping between a predicted 3D shape and the ground truth 3D shape (i.e., the mapping is bijective). The vector field decoder can synthesize 3D shapes in multiple categories and can also synthesize 3D shapes for objects that were not included in the training dataset. In other words, the vector field decoder is also capable of zero-shot generation.

Claims (64)

1 . A computer-implemented method of synthesizing a three-dimensional (3D) shape, comprising:

receiving a latent representation corresponding to the 3D shape;

applying learned parameters to the latent representation, by an implicit decoder neural network, to predict a generative vector field representation of the 3D shape, wherein the generative vector field maps points in 3D space to vectors, such that a closest point on a surface of the 3D shape is intersected by moving from a point of the points along a vector that is mapped to the point; and

during training of the implicit decoder neural network, the learned parameters are updated based on a truncated regression loss function to reduce differences between the generative vector field and a vector field extracted from the 3D shape.

2 . The computer-implemented method of claim 1 , wherein the latent representation comprises one of a latent code or a feature grid.

3 . The computer-implemented method of claim 1 , wherein vectors in the generative vector field that are mapped to points within a threshold distance of the surface contribute to the updated learned parameters.

4 . The computer-implemented method of claim 3 , wherein remaining vectors in the generative vector field that are mapped to points beyond a threshold distance of the surface and having a length that is less than the threshold distance are modified to change the length to equal or greater than the threshold distance.

5 . The computer-implemented method of claim 1 , wherein during training of the implicit decoder neural network, an encoder neural network processes samples of the 3D shape to produce the latent representation.

6 . The computer-implemented method of claim 5 , wherein the samples comprise at least one of a voxel representation of the 3D shape, a grid-sampled vector field representation of the 3D shape, or a point-cloud representation of the 3D shape.

7 . A system, comprising:

a memory that stores learned parameters; and

a processor that is connected to the memory, wherein the processor is configured to synthesize a three-dimensional (3D) shape by:

receiving a latent representation corresponding to the 3D shape;

applying the learned parameters to the latent representation by an implicit decoder neural network to predict a generative vector field representation of the 3D shape, wherein the generative vector field maps points in 3D space to vectors, such that a closest point on a surface of the 3D shape is intersected by moving from a point of the points along a vector that is mapped to the point; and

during training of the implicit decoder neural network, the learned parameters are updated based on a truncated regression loss function to reduce differences between the generative vector field and a vector field extracted from the 3D shape.

8 . A non-transitory computer-readable media storing computer instructions for synthesizing a three-dimensional (3D) shape that, when executed by one or more processors, cause the one or more processors to perform the steps of:

receiving a latent representation corresponding to the 3D shape;

applying learned parameters to the latent representation by an implicit decoder neural network to predict a generative vector field representation of the 3D shape, wherein the generative vector field maps points in 3D space to vectors, such that a closest point on a surface of the 3D shape is intersected by moving from a point of the points along a vector that is mapped to the point; and

during training of the implicit decoder neural network, the learned parameters are updated based on a truncated regression loss function to reduce differences between the generative vector field and a vector field extracted from the 3D shape.

9 . A computer-implemented method of synthesizing a three-dimensional (3D) shape, comprising:

receiving a latent code corresponding to the 3D shape;

processing the latent code by a decoder neural network to produce a latent representation that comprises a feature grid; and

applying learned parameters to the latent representation, by an implicit decoder neural network, to predict a generative vector field representation of the 3D shape, wherein the generative vector field maps points in 3D space to vectors, such that a closest point on a surface of the 3D shape is intersected by moving from a point of the points along a vector that is mapped to the point.

10 . A system, comprising:

a memory that stores learned parameters; and

a processor that is connected to the memory, wherein the processor is configured to synthesize a three-dimensional (3D) shape by:

receiving a latent code corresponding to the 3D shape;

processing the latent code by a decoder neural network to produce a latent representation that comprises a feature grid; and

applying learned parameters to the latent representation, by an implicit decoder neural network, to predict a generative vector field representation of the 3D shape, wherein the generative vector field maps points in 3D space to vectors, such that a closest point on a surface of the 3D shape is intersected by moving from a point of the points along a vector that is mapped to the point.

11 . A computer-implemented method of synthesizing a three-dimensional (3D) shape, comprising:

receiving a latent representation corresponding to the 3D shape;

performing, by an implicit decoder neural network, an interpolation operation on the latent representation to produce a higher resolution latent representation; and

applying the learned parameters to the higher-resolution latent representation by a multilayer perceptron to predict a generative vector field representation of the 3D shape at sub-voxel resolution, wherein the generative vector field representation maps points in 3D space to vectors, such that a closest point on a surface of the 3D shape is intersected by moving from a point of the points along a vector that is mapped to the point.

12 . A system, comprising:

a memory that stores learned parameters; and

a processor that is connected to the memory, wherein the processor is configured to synthesize a three-dimensional (3D) shape by:

receiving a latent representation corresponding to the 3D shape;

performing, by an implicit decoder neural network, an interpolation operation on the latent representation to produce a higher resolution latent representation; and

applying the learned parameters to the higher-resolution latent representation by a multilayer perceptron to predict a generative vector field representation of the 3D shape at sub-voxel resolution, wherein the generative vector field representation maps points in 3D space to vectors, such that a closest point on a surface of the 3D shape is intersected by moving from a point of the points along a vector that is mapped to the point.

13 . A computer-implemented method of synthesizing a three-dimensional (3D) shape, comprising:

receiving a latent representation corresponding to the 3D shape, wherein the 3D shape is associated with a first object category;

applying learned parameters to the latent representation, by an implicit decoder neural network, to predict a generative vector field representation of the 3D shape, wherein the generative vector field maps points in 3D space to vectors, such that a closest point on a surface of the 3D shape is intersected by moving from a point of the points along a vector that is mapped to the point;

receiving a second latent representation corresponding to a second 3D shape associated with a second object category that is different compared with the first object category; and

applying the learned parameters to the second latent representation by the implicit decoder neural network to predict a second generative vector field representation of the second 3D shape.

14 . The computer-implemented method of claim 13 , wherein 3D shapes in the second object category were not used to train the implicit decoder neural network.

15 . A system, comprising:

a memory that stores learned parameters; and

a processor that is connected to the memory, wherein the processor is configured to synthesize a three-dimensional (3D) shape by:

receiving a latent representation corresponding to the 3D shape, wherein the 3D shape is associated with a first object category;

applying learned parameters to the latent representation, by an implicit decoder neural network, to predict a generative vector field representation of the 3D shape, wherein the generative vector field maps points in 3D space to vectors, such that a closest point on a surface of the 3D shape is intersected by moving from a point of the points along a vector that is mapped to the point;

receiving a second latent representation corresponding to a second 3D shape associated with a second object category that is different compared with the first object category; and

applying the learned parameters to the second latent representation by the implicit decoder neural network to predict a second generative vector field representation of the second 3D shape.

16 . A computer-implemented method of synthesizing a three-dimensional (3D) shape, comprising:

processing a text description of the 3D shape by a diffusion neural network model to produce a latent representation; and

applying learned parameters to the latent representation, by an implicit decoder neural network, to predict a generative vector field representation of the 3D shape, wherein the generative vector field maps points in 3D space to vectors, such that a closest point on a surface of the 3D shape is intersected by moving from a point of the points along a vector that is mapped to the point.

17 . The computer-implemented method of claim 16 , wherein at least one of the steps of processing and applying is performed on a server or in a data center to generate the 3D shape and the 3D shape is streamed to a user device.

18 . The computer-implemented method of claim 16 , wherein at least one of the steps of processing and applying is performed within a cloud computing environment.

19 . The computer-implemented method of claim 16 , wherein at least one of the steps of processing and applying is performed for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle.

20 . The computer-implemented method of claim 16 , wherein at least one of the steps of processing and applying is performed on a virtual machine comprising a portion of a graphics processing unit.

21 . A system, comprising:

a memory that stores learned parameters; and

a processor that is connected to the memory, wherein the processor is configured to synthesize a three-dimensional (3D) shape by:

processing a text description of the 3D shape by a diffusion neural network model to produce a latent representation; and

applying learned parameters to the latent representation, by an implicit decoder neural network, to predict a generative vector field representation of the 3D shape, wherein the generative vector field maps points in 3D space to vectors, such that a closest point on a surface of the 3D shape is intersected by moving from a point of the points along a vector that is mapped to the point.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 28, 2023
From: HAO, ZEKUN; LIU, MING-YU; MALLYA, ARUN MOHANRAY
To: NVIDIA CORPORATION
Reel/Frame 064425/0569 →
Continuity (2)
Provisional Application 63431849 · Dec 12, 2022
Related Publication 20240193887A1 · Jun 13, 2024
References Cited (32)
US 20180218473A1 · Hoppert · 2018 [cited by examiner]
US 20180314917A1 · Mehr · 2018 [cited by examiner]
US 20210149022A1 · Kehl · 2021 [cited by examiner]
US 20230169662A1 · Cao · 2023 [cited by examiner]
Rella et al. “Neural Vector Fields for Implicit Surface Representation and Inference.” arXiv e-prints (2022): arXiv-2204. (Year: 2022). [cited by examiner]
Chen, W., et al., “3PSDF: Three-Pole Signed Distance Function for Learning Surfaces with Arbitrary Topologies,” in CVPR, 2022. [cited by applicant]
Chibane, J., et al., “Neural unsigned distance fields for implicit function learning,” in NeurIPS, 2020. [cited by applicant]
Mello Rella, E., et al., “Neural vector fields for surface representation and inference,” arXiv preprint arXiv:2204.06552, 2022. [cited by applicant]
Venkatesh, R., et al., “Deep implicit surface point prediction networks,” in ICCV, 2021. [cited by applicant]
Venkatesh, R., et al., “DUDE: Deep unsigned distance embeddings for hi-fidelity representation of complex 3D surfaces,” arXiv preprint arXiv:2011.02570, 2020. [cited by applicant]
Sangi, A., et al., “TextCraft: Zero-shot generation of high-fidelity and diverse shapes from text,” arXiv preprint arXiv:2211.01427, 2022. [cited by applicant]
Sangi, A., et al., “CLIP-Forge: Towards zero-shot text-to-shape generation,” in CVPR, 2022. [cited by applicant]
Zeng, X., et al., “LION: Latent point diffusion models for 3D shape generation,” arXiv preprint arXiv:2210.06978, 2022. [cited by applicant]
Zheng, X., et al., SDF-Style-GAN: Implicit SDF-Based StyleGAN for 3D Shape Generation, in Computer Graphics Forum, 2022. [cited by applicant]
Hui, K.H., et al., “Neural wavelet-domain diffusion for 3D shape generation,” arXiv preprint arXiv:2209.08725, 2022. [cited by applicant]
Ibing, M., et al., “3D shape generation with grid-based implicit functions,” in CVPR, 2021. [cited by applicant]
Liu, Z., et al., “Towards implicit text-guided 3D shape generation,” in CVPR, 2022. [cited by applicant]
Luo, S., et al., “Diffusion probabilistic models for 3D point cloud generation,” in CVPR, 2021. [cited by applicant]
Mittal, P., et al., “AutoSDF: Shape priors for 3D completion, reconstruction and generation,” in CVPR, 2022. [cited by applicant]
Chang, A., et al., “ShapeNet: An information-rich 3D model repository,” arXiv preprint arXiv:1512.03012, 2015. [cited by applicant]
Chen, et al., “Text2shape: Generating shapes from natural language by learning joint embeddings,” arXiv preprint arXiv:1803.08495, 2018. [cited by applicant]
Collins, J., et al., “ABO: Dataset and benchmarks for real-world 3D object understanding,” in CVPR, 2022. [cited by applicant]
Dhariwal, P., et al.,, “Diffusion models beat gans on image synthesis,” Advances in Neural Information Processing Systems, 34:8780-8794, 2021. [cited by applicant]
He, K., et al., “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in ICCV, 2015. [cited by applicant]
Karras, T., “Maximizing parallelism in the construction of BVHs, octrees, and k-d trees,” in ACM TOG, 2012. [cited by applicant]
Karras, T., et al., “Fast parallel construction of high-quality bounding volume hierarchies,” in HPG, 2013. [cited by applicant]
Karras, T.,, et al., “Elucidating the design space of diffusion-based generative models,” in NeurIPS, 2022. [cited by applicant]
Liu, N., et al., “Compositional visual generation with composable diffusion models,” arXiv preprint arXiv:2206.01714, 2022. [cited by applicant]
Radford, A., et al., “Learning transferable visual models from natural language supervision,” in ICML, 2021. [cited by applicant]
Rombach, R., et al., “High-resolution image synthesis with latent diffusion models,” in CVPR, 2022. [cited by applicant]
Savva, M., et al., “Semantically-enriched 3D models for common-sense knowledge,” in CVPRW, 2015. [cited by applicant]
Shi, W., et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional and neural network,” in CVPR, 2016. [cited by applicant]