IP Library › Granted Patent US 12,541,955
Granted Patent B1
US 12,541,955 · App. 17/686,281 · Granted Feb 3, 2026

Neural network-based noise synthesis

Inventors: Ali Hatamizadeh (Los Angeles, CA); Hongxu Yin (San Jose, CA); Holger Roth (Rockville, MD); Wenqi Li (London, GB); Jan Kautz (Lexington, MA); Daguang Xu (Potomac, MD); Pavlo Molchanov (Mountain View, CA)
Assignee: NVIDIA Corporation
G06V10/7747G06N3/045G06V10/82
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,541,955
App. No.
17/686,281
Granted
Feb 3, 2026
Kind
B1
Abstract

Apparatuses, systems, and techniques are presented to introduce noise or variation into images. In at least one embodiment, one or more neural networks are used to determine noise to be added to one or more images of one or more objects based, at least in part, upon one or more features of the one or more objects.

Claims (33)

1 . A processor, comprising:

one or more circuits to use one or more neural networks to generate one or more images based, at least in part, upon a noise input and a gradient inversion process using gradients for a vision transformer network, wherein to implement the gradient inversion process the one or more circuits are to adjust one or more network parameters of a neural network of the one or more neural networks to enforce similarity between horizontal and vertical edges of adjacent patches for the one or more images to be generated.

2 . The processor of claim 1 , wherein the one or more circuits are further to determine the noise to be input for the gradient inversion process.

3 . The processor of claim 2 , wherein the gradients include one or more network weight gradients calculated during training of the vision transformer network to generate one or more images.

4 . The processor of claim 1 , wherein the one or more circuits are further to adjust the one or more network parameters to attempt to reduce a loss value determined during the gradient inversion process, wherein the loss value is determined according to an image prior loss function, a patch prior loss function to enforce the similarity between horizontal and vertical edges of adjacent patches, and a gradient matching loss function.

5 . The processor of claim 4 , wherein the one or more circuits are further to utilize a scheduler to adjust weightings of at least one of the image prior loss function and the gradient matching loss function during the gradient inversion process.

6 . The processor of claim 4 , wherein the one or more circuits are further to utilize another neural network of the one or more neural networks to provide an image prior for the image prior loss function during the gradient inversion process.

7 . A system, comprising:

one or more processors to use one or more neural networks to generate one or more images based, at least in part, upon a noise input and a gradient inversion process using gradients for a vision transformer network, wherein to implement the gradient inversion process the one or more processors are to adjust one or more network parameters of a neural network of the one or more neural networks to enforce similarity between horizontal and vertical edges of adjacent patches for the one or more images to be generated.

8 . The system of claim 7 , wherein the one or more processors are further to determine the noise to be input for the gradient inversion process.

9 . The system of claim 8 , wherein the gradients include one or more network weight gradients calculated during training of the vision transformer network to generate one or more images.

10 . The system of claim 7 , wherein the one or more processors are further to adjust the one or more network parameters to attempt to reduce a loss value determined during the gradient inversion process, wherein the loss value is determined according to an image prior loss function, a patch prior loss function to enforce the similarity between horizontal and vertical edges of adjacent patches, and a gradient matching loss function.

11 . The system of claim 10 , wherein the one or more processors are further to utilize a scheduler to adjust weightings of at least one of the image prior loss function and the gradient matching loss function during the gradient inversion process.

12 . The system of claim 10 , wherein the one or more processors are further to utilize another neural network of the one or more neural networks to provide an image prior for the image prior loss function during the gradient inversion process.

13 . A method, comprising:

using one or more neural networks to generate one or more images based, at least in part, upon a noise input and a gradient inversion process using gradients for a vision transformer network, the gradient inversion process comprising adjusting one or more network parameters of a neural network of the one or more neural networks to enforce similarity between horizontal and vertical edges of adjacent patches for the one or more images to be generated.

14 . The method of claim 13 , further comprising:

determining the noise to be input for the gradient inversion process.

15 . The method of claim 14 , wherein the gradients include one or more network weight gradients calculated during training of the vision transformer network to generate one or more images.

16 . The method of claim 13 , further comprising:

adjusting the one or more network parameters to attempt to reduce a loss value determined during the gradient inversion process, wherein the loss value is determined according to an image prior loss function, a patch prior loss function to enforce the similarity between horizontal and vertical edges of adjacent patches, and a gradient matching loss function.

17 . The method of claim 16 , further comprising:

utilizing a scheduler to adjust weightings of at least one of the image prior loss function and the gradient matching loss function during the gradient inversion process.

18 . The method of claim 16 , further comprising:

utilizing another neural network of the one or more neural networks to provide an image prior for the image prior loss function during the gradient inversion process.

19 . An image generation system, comprising:

one or more processors to use one or more neural networks to generate one or more images based, at least in part, upon a noise input and a gradient inversion process using gradients for a vision transformer network, wherein to implement the gradient inversion process the one or more processors are to adjust one or more network parameters of a neural network of the one or more neural networks to enforce similarity between horizontal and vertical edges of adjacent patches for the one or more images to be generated; and

memory for storing network parameters for the one or more neural networks.

20 . The image generation system of claim 19 , wherein the one or more processors are further to determine the noise to be input for the gradient inversion process.

21 . The image generation system of claim 20 , wherein the gradients include one or more network weight gradients calculated during training of vision transformer neural network to generate one or more images.

22 . The image generation system of claim 19 , wherein the one or more processors are further to adjust the one or more network parameters to attempt to reduce a loss value determined during the gradient inversion process, wherein the loss value is determined according to an image prior loss function, a patch prior loss function to enforce the similarity between horizontal and vertical edges of adjacent patches, and a gradient matching loss function.

23 . The image generation system of claim 22 , wherein the one or more processors are further to utilize a scheduler to adjust weightings of at least one of the image prior loss function and the gradient matching loss function during the gradient inversion process.

24 . The image generation system of claim 22 , wherein the one or more processors are further to utilize another neural network of the one or more neural networks to provide an image prior for the image prior loss function during the gradient inversion process.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 29, 2022
From: HATAMIZADEH, ALI; YIN, HONGXU; ROTH, HOLGER; LI, WENQI; KAUTZ, JAN; XU, DAGUANG; MOLCHANOV, PAVLO
To: NVIDIA CORPORATION
Reel/Frame 059430/0723 →
References Cited (53)
US 20220172050A1 · Dalli · 2022 [cited by examiner]
Forssell, M. (2014). Hardware implementation of artificial neural networks. Information flow in networks, 18(2014), 1-4. (Year: 2014). [cited by examiner]
Mordvintsev. (Aug. 12, 2015). Deepdream/dream.ipynb at master ⋅ Google/deepdream, Inceptionism. GitHub. https://github.com/google/deepdream/blob/master/dream.ipynb (Year: 2015). [cited by examiner]
Bau, D., Strobelt, H., Peebles, W., Wulff, J., Zhou, B., Zhu, J. Y., & Torralba, A. (2020). Semantic photo manipulation with a generative image prior. arXiv preprint arXiv:2005.07727. (Year: 2020). [cited by examiner]
Lopez-Alvis J, Laloy E, Nguyen F, Hermans T. Deep generative models in inversion: a review and development of a new approach based on a variational autoencoder. arXiv preprint arXiv:2008.12056. Aug. 27, 2020. (Year: 202… [cited by examiner]
Lu, J., Zhang, X. S., Zhao, T., He, X., & Cheng, J. (2021). April: Finding the Achilles' Heel on Privacy for Vision Transformers. CoRR, abs/2112.14087. Retrieved from https://arxiv.org/abs/2112.14087 (Year: 2021). [cited by examiner]
Jeon, J., Lee, K., Oh, S., & Ok, J. (2021). Gradient inversion with generative image prior. Advances in neural information processing systems, 34, 29898-29908. (Year: 2021). [cited by examiner]
Yin H, Mallya A, Vahdat A, Álvarez JM, Kautz J, Molchanov P. See Through Gradients: Image Batch Recovery via GradInversion. InCVPR Jan. 1, 2021. (Year: 2021). [cited by examiner]
Brock et al., “Large Scale GAN Training for High Fidelity Natural Image Synthesis,” ICLR, Feb. 25, 2019, 35 pages. [cited by applicant]
Cai et al., “ZeroQ: A Novel Zero Shot Quantization Framework,” CVPR, 2020, 10 pages. [cited by applicant]
Chen et al., “Data-Free Learning of Student Networks,” Proceedings of the IEEE International Conference on Computer Vision, 2019, 10 pages. [cited by applicant]
Chen et al., “Improved Baselines with Momentum Contrastive Learning,” Mar. 9, 2020, 3 pages. [cited by applicant]
Cheng et al., “Per-Pixel Classification is Not All You Need for Semantic Segmentation,” Oct. 31, 2021, 17 pages. [cited by applicant]
Dai et al., “UP-DETR: Unsupervised Pre-training for Object Detection with Transformers,” Apr. 7, 2021, 11 Pages. [cited by applicant]
Deng et al., “Imagenet: A Large-Scale Hierarchical Image Database,” CVPR, 2009, 8 pages. [cited by applicant]
Dosovitskiy et al., “An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale,” Oct. 22, 2020, 21 pages. [cited by applicant]
Geiping et al., “Inverting Gradients—How Easy is it to Break Privacy in Federated Learning?,” Neural Information Processing System, 2020, 11 pages. [cited by applicant]
Gulrajani et al., “Improved Training of Wasserstein GANs,” Advances in Neural Information Processing Systems, 2017, 17 pages. [cited by applicant]
Guo et al., “MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition,” European Conference on Computer Vision, Jul. 27, 2016, 17 pages. [cited by applicant]
He et al., “Deep Residual Learning for Image Recognition,” CVPR, 2016, 9 pages. [cited by applicant]
He et al., “Momentum Contrast for Unsupervised Visual Representation Learning,” CVPR, 2020, 10 pages. [cited by applicant]
Ieee, “IEEE Standard 754-2008 (Revision of IEEE Standard 754-1985): IEEE Standard for Floating-Point Arithmetic,” Aug. 29, 2008, 70 pages. [cited by applicant]
Jeon et al., “Gradient Inversion with Generative Image Prior,” Oct. 28, 2021, 17 pages. [cited by applicant]
Jin et al., “CAFE: Catastrophic Data Leakage in Vertical Federated Learning,” Oct. 26, 2021, 23 pages. [cited by applicant]
Karras et al., “Analyzing and Improving the Image Quality of Stylegan,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, 10 pages. [cited by applicant]
Kingma et al. “Adam: A Method for Stochastic Optimization,” Dec. 22, 2014, 9 pages. [cited by applicant]
Kurmi et al., “Domain Impression: A Source Data Free Domain Adaptation Method,” Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, 11 pages. [cited by applicant]
Lecun et al., “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE, 86(11): 1998, 47 pages. [cited by applicant]
Liu et al., “Source-Free Domain Adaptation for Semantic Segmentation,” CVPR, 2021, 10 pages. [cited by applicant]
Luo et al., “Large-Scale Generative Data-Free Distillation,” Dec. 10, 2020, 12 pages. [cited by applicant]
Mahendran et al., “Understanding Deep Image Representations by Inverting Them,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 9 pages. [cited by applicant]
Mahendran et al., “Visualizing Deep Convolutional Neural Networks using Natural Pre-Images,” IJCV, Apr. 14, 2016, 25 pages. [cited by applicant]
McMahan et al., “Communication Efficient Learning of Deep Networks from Decentralized Data,” Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, 2017, 10 pages. [cited by applicant]
Melis et al., “Exploiting Unintended Feature Leakage in Collaborative Learning,” IEEE Symposium on Security & Privacy, Nov. 1, 2018, 16 pages. [cited by applicant]
Miyato et al., “Spectral Normalization for Generative Adversarial Networks,” ICLR, 2018, 26 pages. [cited by applicant]
Mordvintsev et al., Inceptionism: Going Deeper Into Neural Networks, retrieved from https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html, Jun. 17, 2015, 6 pages. [cited by applicant]
Nguyen et al., “Plug & Play Generative Networks: Conditional Iterative Generation of Images in Latent Space,” CVPR, 2017, 11 pages. [cited by applicant]
Nguyen et al., “Synthesizing the Preferred Inputs for Neurons in Neural Networks via Deep Generator Networks,” Advances in Neural Information Processing Systems, 2016, 9 pages. [cited by applicant]
Raghu et al., “Do Vision Transformers See Like Convolutional Neural Networks?” Aug. 19, 2021, 29 pages. [cited by applicant]
Santurkar et al., “Image Synthesis with a Single (Robust) Classifier,” Neural Information Processing System, 2019, 12 pages. [cited by applicant]
Shen et al., “RANSAC-Flow: Generic Two-Stage Image Alignment,” ECCV, 2020, 20 pages. [cited by applicant]
Shokri et al., “Membership Inference Attacks Against Machine Learning Models,” IEEE Security and Privacy Workshops, 2017, 7 pages. [cited by applicant]
Society of Automotive Engineers On-Road Automated Vehicle Standards Committee, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Standard No. J3016-201609, issued Jan… [cited by applicant]
Society of Automotive Engineers On-Road Automated Vehicle Standards Committee, “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” Standard No. J3016-201806, issued Jan… [cited by applicant]
Touvron et al., “Training Data-Efficient Image Transformers & Distillation Through Attention,” International Conference on Machine Learning, 2021, 22 pages. [cited by applicant]
Wang et al., “Beyond Inferring Class Representatives: User-Level Privacy Leakage from Federated Learning,” INFOCOM, Dec. 5, 2018, 10 pages. [cited by applicant]
Yin et al., “Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion,” CVPR, 2020, 10 pages. [cited by applicant]
Yin et al., “See Through Gradients: Image Batch Recovery via Gradinversion,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, 10 pages. [cited by applicant]
Zhai et al., “Scaling Vision Transformers,” Jun. 8, 2021, 31 Pages. [cited by applicant]
Zhang et al., “Self-Attention Generative Adversarial Networks,” ICML, 2019, 10 pages. [cited by applicant]
Zhang et al., “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,” CVPR, 2018, 10 pages. [cited by applicant]
Zhong et al., “Face Transformer for Recognition,” Apr. 13, 2021, 5 pages. [cited by applicant]
Zhu et al., “Deep Leakage from Gradients,” Neural Information Processing System, 2019, 11 pages. [cited by applicant]
Cited By (1)
US 12,731,298