IP Library › Granted Patent US 12,223,611
Granted Patent B2
US 12,223,611 · App. 18/149,609 · Granted Feb 11, 2025

Generating styles for neural style transfer in three-dimensional shapes

Inventors: Hooman Shayani (Longfield, GB); Marco Fumero (Rome, IT); Aditya Sanghi (Toronto, CA)
Assignee: AUTODESK, INC.
G06T19/20G06N3/0455G06N3/0475G06N3/08G06N3/092G06T17/00G06T17/10G06T2210/56G06T2219/2021G06T2219/2024
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,223,611
App. No.
18/149,609
Granted
Feb 11, 2025
Kind
B2
Abstract

One embodiment of the present invention sets forth a technique for performing style transfer. The technique includes determining a distribution associated with a plurality of style codes for a plurality of three-dimensional (3D) shapes, where each style code included in the plurality of style codes represents a difference between a first 3D shape and a second 3D shape, and where the second 3D shape is generated by applying one or more augmentations to the first 3D shape. The technique also includes sampling from the distribution to generate an additional style code and executing a trained machine learning model based on the additional style code to generate an output 3D shape having style-based attributes associated with the additional style code and content-based attributes associated with an object. The technique further includes generating a 3D model of the object based on the output 3D shape.

Claims (41)

1. A computer-implemented method for performing style transfer, the method comprising:

determining a distribution associated with a plurality of style codes for a plurality of three-dimensional (3D) shapes, wherein each style code included in the plurality of style codes represents a difference between a first 3D shape and a second 3D shape, and wherein the second 3D shape is generated by applying one or more augmentations to the first 3D shape;

sampling from the distribution to generate an additional style code;

executing a first trained machine learning model based on the additional style code to generate an output 3D shape having one or more style-based attributes associated with the additional style code and one or more content-based attributes associated with an object; and

generating a 3D model of the object based on the output 3D shape.

2. The computer-implemented method of claim 1 , wherein determining the distribution comprises training a second machine learning model to learn the distribution based on one or more losses associated with the plurality of style codes.

3. The computer-implemented method of claim 2 , wherein the second machine learning model comprises a generative model.

4. The computer-implemented method of claim 1 , wherein sampling from the distribution comprises converting a randomized input into the additional style code.

5. The computer-implemented method of claim 1 , wherein sampling from the distribution comprises generating the additional style code based on a text-based prompt associated with the output 3D shape.

6. The computer-implemented method of claim 1 , wherein sampling from the distribution comprises:

sampling from a first distribution to generate a first portion of the additional style code that represents a first subset of the one or more style-based attributes; and

sampling from a second distribution to generate a second portion of the additional style code that represents a second subset of the one or more style-based attributes.

7. The computer-implemented method of claim 1 , wherein generating the output 3D shape comprises:

inputting the additional style code and an input 3D shape associated with the object into a decoder neural network included in the first trained machine learning model; and

executing the decoder neural network to generate the output 3D shape.

8. The computer-implemented method of claim 7 , wherein the output 3D shape comprises at least one of a set of signed distance function values or a set of occupancy values.

9. The computer-implemented method of claim 1 , wherein sampling from the distribution comprises generating the additional style code based on an aggregation of two or more style codes included in the plurality of style codes.

10. The computer-implemented method of claim 9 , wherein the aggregation comprises at least one of an average, a weighted average, or an interpolation.

11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:

determining a distribution associated with a plurality of style codes for a plurality of three-dimensional (3D) shapes, wherein each style code included in the plurality of style codes represents a difference between a first 3D shape and a second 3D shape, and wherein the second 3D shape is generated by applying one or more augmentations to the first 3D shape;

sampling from the distribution to generate an additional style code;

executing a first trained machine learning model based on the additional style code to generate an output 3D shape having one or more style-based attributes associated with the additional style code and one or more content-based attributes associated with an object; and

generating a 3D model of the object based on the output 3D shape.

12. The one or more non-transitory computer-readable media of claim 11 , wherein determining the distribution comprises training a second machine learning model to learns the distribution based on one or more losses associated with the plurality of style codes.

13. The one or more non-transitory computer-readable media of claim 12 , wherein the one or more losses comprise a contrastive loss that is generated based on the plurality of style codes and a plurality of additional inputs associated with the plurality of style codes.

14. The one or more non-transitory computer-readable media of claim 11 , wherein sampling from the distribution comprises converting, via a second trained machine learning model, one or more input values into the additional style code.

15. The one or more non-transitory computer-readable media of claim 14 , wherein the one or more input values comprise at least one of a randomized input, one or more style codes included in the plurality of style codes, or a portion of a style code.

16. The one or more non-transitory computer-readable media of claim 14 , wherein the one or more input values comprise at least one of a description of the one or more style-based attributes or a representation of the one or more style-based attributes.

17. The one or more non-transitory computer-readable media of claim 11 , wherein determining the distribution associated with the plurality of style codes comprises:

determining a first distribution associated with a first portion of the plurality of style codes that represents a first level of detail associated with the one or more style-based attributes; and

determining a second distribution associated with a second portion of the plurality of style codes that represents a second level of detail associated with the one or more style-based attributes.

18. The one or more non-transitory computer-readable media of claim 11 , wherein sampling from the distribution comprises generating the additional style code based on an aggregation of two or more style codes included in the plurality of style codes.

19. The one or more non-transitory computer-readable media of claim 11 , wherein the one or more augmentations comprise at least one of a smoothing augmentation or a coarsening augmentation.

20. A system, comprising:

one or more memories that store instructions, and

one or more processors that are coupled to the one or more memories and,

when executing the instructions, are configured to perform the steps of:

determining a distribution associated with a plurality of style codes for a plurality of three-dimensional (3D) shapes, wherein each style code included in the plurality of style codes represents a difference between a first 3D shape and a second 3D shape, and wherein the second 3D shape is generated by applying one or more augmentations to the first 3D shape;

sampling from the distribution to generate an additional style code;

executing a first trained machine learning model based on the additional style code to generate an output 3D shape having one or more style-based attributes associated with the additional style code and one or more content-based attributes associated with an object; and

generating a 3D model of the object based on the output 3D shape.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 3, 2023
From: SHAYANI, HOOMAN; FUMERO, MARCO; SANGHI, ADITYA
To: AUTODESK, INC.
Reel/Frame 062880/0408 →
Continuity (2)
Provisional Application 63328658 · Apr 7, 2022
Related Publication 20230326159A1 · Oct 12, 2023
References Cited (48)
US 20040135788A1 · Davidson et al. · 2004 [cited by applicant]
US 20180373999A1 · Xu · 2018 [cited by applicant]
US 20190244329A1 · Li et al. · 2019 [cited by applicant]
US 20200151559A1 · Karras · 2020 [cited by examiner]
US 20210183033A1 · Chung et al. · 2021 [cited by applicant]
US 20210264236A1 · Xu et al. · 2021 [cited by applicant]
US 20210279938A1 · Chandran et al. · 2021 [cited by applicant]
US 20220156415A1 · Meltzer et al. · 2022 [cited by applicant]
US 20220156416A1 · Meltzer et al. · 2022 [cited by applicant]
US 20220156420A1 · Meltzer et al. · 2022 [cited by applicant]
US 20220156987A1 · Chandran · 2022 [cited by examiner]
US 20230109732A1 · Hertzmann et al. · 2023 [cited by applicant]
Berkiten et al., “Learning Detail Transfer based on Geometric Features”, DOI:10.1111/cgf.13132, vol. 36, No. 2, 2017, pp. 361-373. [cited by applicant]
Cao et al., “PSNet: A Style Transfer Network for Point Cloud Stylization on Geometry and Color”, IEEE, https://doi.org/10.1109/WACV45572.2020.9093513, 2020, pp. 3326-3334. [cited by applicant]
Chang et al., “ShapeNet: An Information-Rich 3D Model Repository”, https://arxiv.org/abs/1512.03012, Dec. 9, 2015, 11 pages. [cited by applicant]
Chen et al., “DECOR-GAN: 3D Shape Detailization by Conditional Refinement”, arXiv:2012.09159, Mar. 29, 2021, 25 pages. [cited by applicant]
Chen et al., “Learning Implicit Fields for Generative Shape Modeling”, arXiv:1812.02822, Sep. 16, 2019, 10 pages. [cited by applicant]
Chen et al., “Multiresolution Deep Implicit Functions for 3D Shape Representation”, https://arxiv.org/abs/2109.05591, Sep. 2021, 20 pages. [cited by applicant]
Chibane et al., “Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion”, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6970-6981. [cited by applicant]
Gatys et al., “Image Style Transfer Using Convolutional Neural Networks”, In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2016, pp. 2414-2423. [cited by applicant]
Gropp et al., “Implicit Geometric Regularization for Learning Shapes”, ICML, arXiv:2002.10099, Jul. 9, 2020, 14 pages. [cited by applicant]
Huang et al., “Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization”, arXiv:1703.06868, Jul. 30, 2017, 11 pages. [cited by applicant]
Jing et al., “Neural Style Transfer: A Review”, https://arxiv.org/abs/1705.04058, May 11, 2017, 14 pages. [cited by applicant]
Koch et al., “ABC: A Big CAD Model Dataset for Geometric Deep Learning”, https://arxiv.org/abs/1812.06216, Dec. 15, 2018, 15 pages. [cited by applicant]
Kolkin et al., “Style Transfer by Relaxed Optimal Transport and Self-Similarity”, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 10051-10060. [cited by applicant]
Li et al., “Demystifying Neural Style Transfer”, arXiv:1701.01036, Jul. 1, 2017, 7 pages. [cited by applicant]
Liu et al., “Cubic Stylization”, https://doi.org/10.1145/3355089.3356495, ACM Transactions on Graphics, vol. 38, No. 6, Article 197, Nov. 2019, pp. 197:1-197:10. [cited by applicant]
Liu et al., “Normal-Driven Spherical Shape Analogies”, https://arxiv.org/abs/2104.11993, vol. 40, No. 5, Jul. 11, 2021, 11 pages. [cited by applicant]
Liu et al., “Neural Subdivision”, ACM Transactions on Graphics, vol. 39, No. 4, Article 1, https:/ /doi.org/10.1145/3386569.3392418, Jul. 2020, pp. 1:1-1:16. [cited by applicant]
Liu et al., “A Survey of Manufacturing Oriented Topology Optimization Methods”, https://doi.org/10.1016/j.advengsoft.2016.07.017, Advances in Engineering Software, vol. 100, 2016, pp. 161-175. [cited by applicant]
Lun et al., “Elements of Style: Learning Perceptual Shape Style Similarity”, https://doi.org/10.1145/2766929, ACM Transactions on Graphics, vol. 34, No. 4, Article 84, Aug. 2015, pp. 84:1-84:14. [cited by applicant]
Lun et al., “Functionality Preserving Shape Style Transfer”, https://doi.org/10.1145/2980179.2980237, ACM Transactions on Graphics, vol. 35, No. 6, Article 209, Nov. 2016, pp. 209:1-209:14. [cited by applicant]
Ma et al., “Analogy-Driven 3D Style Transfer”, In Computer Graphics Forum, vol. 33, No. 2, 2014, pp. 175-184. [cited by applicant]
Marin et al., “Instant Recovery of Shape from Spectrum via Latent Space Connections”, arXiv:2003.06523, Nov. 4, 2020, 17 pages. [cited by applicant]
Meltzer et al., “UVStyle-Net: Unsupervised Few-shot Learning of 3D Style Similarity Measure for B-Reps”, arXiv:2105.02961, Oct. 11, 2021, 13 pages. [cited by applicant]
Mescheder et al., “Occupancy Networks: Learning 3D Reconstruction in Function Space”, arXiv:1812.03828, Apr. 30, 2019, 11 pages. [cited by applicant]
Mroueh, Youssef, “Wasserstein Style Transfer”, In International Conference on Artificial Intelligence and Statistics, PMLR, vol. 108, 2020, 10 pages. [cited by applicant]
Park et al., “DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation”, arXiv:1901.05103, Jan. 16, 2019, 19 pages. [cited by applicant]
Qi et al., “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation”, https://arxiv.org/abs/1612.00593, Dec. 2, 2016, 19 pages. [cited by applicant]
Segu et al., “3DSNet: Unsupervised Shape-to-Shape 3D Style Transfer”, arXiv:2011.13388, Nov. 26, 2020, 14 pages. [cited by applicant]
Sorkine, Olga, “Laplacian Mesh Processing”, https://doi.org/10.2312/egst.20051044, Eurographics, 2005, 18 pages. [cited by applicant]
Wang et al., “Intrinsic and Extrinsic Operators for Shape Analysis”, Handbook of Numerical Analysis, Chapter 2, vol. 20, 2019, pp. 41-115. [cited by applicant]
Barroqueiro et al., “Bridging between topology optimization and additive manufacturing via Laplacian smoothing”, Journal of Mechanical Design, vol. 143, No. 9, 2021, 9 pages. [cited by applicant]
Notice of Allowance received for U.S. Appl. No. 18/149,605 dated Aug. 7, 2024, 17 pages. [cited by applicant]
Notice of Allowance received for U.S. Appl. No. 18/149,601 dated Aug. 19, 2024, 17 pages. [cited by applicant]
Friedrich et al., “Standardization of Gram Matrix for Improved 3D Neural Style Transfer”, IEEE Symposium Series on Computational Intelligence, Dec. 6-9, 2019, pp. 1375-1382. [cited by applicant]
Regateiro et al., “3D Human Shape Style Transfer”, Computer Vision and Pattern Recognition, arXiv:2109.01587, Sep. 6, 2021, pp. 1-12. [cited by applicant]
Xie et al., “Design of Painting Art Style Rendering System Based on Convolutional Neural Network”, Scientific Programming, https://doi.org/10.1155/2021/4708758, 2021, pp. 1-21. [cited by applicant]