IP Library › Granted Patent US 12,223,706
Granted Patent B2
US 12,223,706 · App. 17/824,694 · Granted Feb 11, 2025

Training energy-based models from a single image for internal learning and inference using trained models

Inventors: Zilong Zheng (Beijing, CN); Jianwen Xie (Santa Clara, CA); Ping Li (Bellevue, WA)
Assignee: Baidu USA LLC
G06V10/85G06V10/774G06V10/82
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,223,706
App. No.
17/824,694
Granted
Feb 11, 2025
Kind
B2
Abstract

Different from prior works that model the internal distribution of patches within an image implicitly with a top-down latent variable model (e.g., generator), embodiments explicitly represent the statistical distribution within a single image by using an energy-based generative framework, where a pyramid of energy functions, each parameterized by a bottom-up deep neural network, are used to capture the distributions of patches at different resolutions. Also, embodiments of a coarse-to-fine sequential training and sampling strategy are presented to train the model efficiently. Besides learning to generate random samples from white noise, embodiments can learn in parallel with a self-supervised task (e.g., recover an input image from its corrupted version), which can further improve the descriptive power of the learned model. Embodiments does not require an auxiliary model (e.g., discriminator) to assist the training, and embodiments also unify internal statistics learning and image generation in a single framework.

Claims (46)

1. A computer-implemented method comprising:

given a set of energy-based neural network models (EBMs) in which each EBM is configured to synthesize an image at a scale and the scales of the set of EBMs range from a minimal scale to a maximum scale, and given a multi-scale set of versions of a training image that similarly range from the minimal scale to the maximum scale so that each EBM has a corresponding version of the training image at the same scale as the EBM, performing steps comprising:

training each of the EBMs in the set of EBMs to synthesize an image at the scale of the EBM based on a patch distribution learned from the version of the training image at the scale of the EBM given an initial input image at the scale of the EBM and the version of the training image at the scale of the EBM, in which for at least some of the EBMs the initial input image is a synthesized image from a lower scaled EBM that has been upsampled to the scale of the EBM, comprising:

for each EBM:

using the initial input image for the EBM and the EBM in a Markov chain Monte Carlo (MCMC) sampling process to produce the synthesized image at the scale of the EBM; and

updating parameters of the EBM by using a comparison comprising one or more the synthesized image generated using the MCMC sampling process and the version of the training image at the scale of the EBM; and

responsive to a training stop condition being met, output the trained set of EBMs.

2. The computer-implemented method of claim 1 wherein the MCMC process is an iterative process with a finite number of Langevin dynamics steps.

3. The computer-implemented method of claim 2 wherein the finite number of Langevin dynamics steps for the MCMC sample process is set for each EBM and is a same or a different finite number than one or more for other EBMs.

4. The computer-implemented method of claim 2 wherein the synthesized image from a lower scaled EBM that has been upsampled to the scale of the EBM is the synthesized image generated at a last step of the finite number of Langevin dynamics steps.

5. The computer-implemented method of claim 1 wherein, for the EBM with the minimal scale, the initial input image is an image generated by randomly sampling from a uniform distribution.

6. The computer-implemented method of claim 1 wherein, the comparison comprising one or more the synthesized image generated by the EBM using the MCMC sampling process and the version of the training image at the scale of the EBM comprises:

determining a loss gradient that at least approximates difference between internal statistics of the training image as determined the EBM versus an average of internal statistics of one or more synthesized images as determined by the EBM.

7. The computer-implemented method of claim 1 further comprising, given an input condition image:

additionally training the set of EBMs for a conditional task using the input condition image as an input to the EBM with the minimum scale, in which training loss for the set of EBMs represents a combination of unconditional generation loss and conditional generation loss.

8. The computer-implemented method of claim 7 wherein:

the input condition image is a low-resolution image and the conditional task is super-resolution;

the input condition image is a noisy image and the conditional task is denoising; and

the input condition image is an image from another domain and the conditional task is cross-domain image translation.

9. A computer-implemented method comprising:

obtaining the trained set of (EBMs) trained according to the method of claim 1 ;

starting with a trained EBM with the minimum scale and moving in order of increasing scale until the trained EBM with the maximum scale has been processed, performing steps comprising:

responsive to the trained EBM being the trained EBM with the minimum scale, using an initial input image having the minimum scale and the trained EBM in a Markov chain Monte Carlo (MCMC) sampling process to produce a final synthesized image at the scale of the trained EBM; and

responsive to the trained EBM not being the trained EBM with the minimum scale, using an initial image having the scale of the trained EBM and the trained EBM in a Markov chain Monte Carlo (MCMC) sampling process to produce a final synthesized image at the scale of the trained EBM, in which the initial image is obtained from the final synthesized image from the trained EBM with a lower scale; and

outputting at least one of the final synthesized images.

10. The computer-implemented method of claim 9 the MCMC process is an iterative process with a finite number of Langevin dynamics steps.

11. The computer-implemented method of claim 9 wherein, for the trained EBM with the minimal scale, the initial input image is one of:

an image generated by randomly sampling from a uniform distribution;

a low-resolution image;

a noisy image; or

an image from domain different from that used to train the trained set of EBMs.

12. A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising:

given a set of energy-based neural network models (EBMs) in which each EBM is configured to synthesize an image at a scale and the scales of the set of EBMs range from a minimal scale to a maximum scale, and given a multi-scale set of versions of a training image that similarly range from the minimal scale to the maximum scale so that each EBM has a corresponding version of the training image at the same scale as the EBM, performing steps comprising:

training each of the EBMs in the set of EBMs to synthesize an image at the scale of the EBM based on a patch distribution learned from the version of the training image at the scale of the EBM given an initial input image at the scale of the EBM and the version of the training image at the scale of the EBM, in which for at least some of the EBMs the initial input image is a synthesized image from a lower scaled EBM that has been upsampled to the scale of the EBM, comprising:

for each EBM:

using the initial input image for the EBM and the EBM in a Markov chain Monte Carlo (MCMC) sampling process to produce the synthesized image at the scale of the EBM; and

updating parameters of the EBM by using a comparison comprising one or more the synthesized image generated using the MCMC sampling process and the version of the training image at the scale of the EBM; and

responsive to a training stop condition being met, output the trained set of EBMs.

13. The non-transitory computer-readable medium or media of claim 12 wherein the MCMC process is an iterative process with a finite number of Langevin dynamics steps and wherein the finite number of Langevin dynamics steps for the MCMC sample process is a same or a different finite number than for one or more other EBMs.

14. The non-transitory computer-readable medium or media of claim 13 wherein the synthesized image from a lower scaled EBM that has been upsampled to the scale of the EBM is the synthesized image generated at a last step of the finite number of Langevin dynamics steps.

15. The non-transitory computer-readable medium or media of claim 12 wherein, for the EBM with the minimal scale, the initial input image is an image generated by randomly sampling from a uniform distribution.

16. The non-transitory computer-readable medium or media of claim 13 wherein, the comparison comprising one or more the synthesized image generated by the EBM using the MCMC sampling process and the version of the training image at the scale of the EBM comprises:

determining a loss gradient that at least approximates difference between internal statistics of the training image as determined the EBM versus an average of internal statistics of one or more synthesized images as determined by the EBM.

17. The non-transitory computer-readable medium or media of claim 12 further comprising one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising, given an input condition image:

additionally training the set of EBMs for a conditional task using the input condition image as an input to the EBM with the minimum scale, in which training loss for the set of EBMs represents a combination of unconditional generation loss and conditional generation loss.

18. The non-transitory computer-readable medium or media of claim 12 wherein each of the EBMs is a patchwise generative convolution neural network.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 21, 2022
From: ZHENG, ZILONG; XIE, JIANWEN; LI, PING
To: BAIDU USA LLC
Reel/Frame 062172/0237 →
Continuity (2)
Provisional Application 63208842 · Jun 9, 2021
Related Publication 20220398836A1 · Dec 15, 2022
References Cited (48)
Shaham et al., SinGAN: Learning a Generative Model from a Single Natural Image, 2019, Proc. of the IEEE Conf. on Computer Vision& Pattern Recognition. (Year: 2019). [cited by examiner]
Xie et al., Cooperative Learning of Energy-Based Model and Latent Variable Model via MCMC Teaching, 2018, The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18). (Year: 2018). [cited by examiner]
Xie et al.,“Generative VoxelNet:learning energy-based models for 3D shape synthesis and analysis,” arXiv preprint arXiv:2012.13522, 2020. (16pgs). [cited by applicant]
Xie et al.,“Learning energybased model with variational auto-encoder as amortized sampler,” arXiv preprint arXiv:2012.14936, 2021. (11pgs). [cited by applicant]
Xie et al.,“Synthesizing dynamic patterns by spatial-temporal generative convnet,” arXiv preprint arXiv:1606.00972, 2017. (9pgs). [cited by applicant]
Xie et al.,“Learning energy-based spatial-temporal generative convnets for dynamic patterns,” arXiv preprint arXiv:1909.11975, 2019. (15pgs). [cited by applicant]
Yu et al.,“LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop,” arXiv preprint arXiv:1506.03365, 2016. (9 pgs). [cited by applicant]
Zhou et al.,“Learning deep features for scene recognition using places database,” In Advances in Neural Information Processing Systems (NIPS), 2014. (9 pgs). [cited by applicant]
Zhu et al.,“Exploring texture ensembles by efficient Markov chain Monte Carlo-toward a ‘trichromacy’ theory of texture,” IEEE Transactions on Pattern Analysis & Machine Intelligence (TPAMI), 2000. (17pgs). [cited by applicant]
Zhu et al.,“Filters, Random Fields and Maximum Entropy (FRAME) Towards a unified theory for texture modeling,” International Journal of Computer Vision (IJCV), 27(2):107-126, 1998. (44pgs). [cited by applicant]
Goodfellow et al.,“Generative adversarial nets,” arXiv preprint arXiv: 1406.2661, 2014. (9pgs). [cited by applicant]
Grathwohl et al.,“Your classifier is secretly an energy based model and you should treat it like one,” arXiv preprint arXiv: 1912.03263, 2020. (23pgs). [cited by applicant]
Han et al.,“Divergence triangle for joint training of generator model, energy-based model, and inferential model,” arXiv preprint arXiv:1812.10907, 2019. (13pgs). [cited by applicant]
Heusel et al.,“Gans trained by a two time-scale update rule converge to a local nash equilibrium,” arXiv preprint arXiv:1706.08500, 2018. (38pgs). [cited by applicant]
Isola et al.,“Image-to-image translation with conditional adversarial networks,” arXiv preprint arXiv:1611.07004, 2018. (17pgs). [cited by applicant]
Bela Julesz, “Visual pattern discrimination,” IRE transactions on Information Theory, 8(2):84-92, 1962. [Abstract] (1pg). [cited by applicant]
Kingma et al.,“Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2017. (15pgs). [cited by applicant]
Kingma et al.,“Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2014. (14pgs). [cited by applicant]
LeCun et al.,“A tutorial on energy-based learning,” Predicting structured data, 1(0), 2006.(60pgs). [cited by applicant]
Li et al.,“Precomputed real-time texture synthesis with markovian generative adversarial networks,” arXiv preprint arXiv:1604.04382, 2016. (17pgs). [cited by applicant]
Jun S Liu, “Monte Carlo Strategies in Scientific Computing,” Springer Science & Business Media, 2008. (7pgs). [cited by applicant]
Marr et al.,“A computational theory of human stereo vision,” Proceedings of the Royal Society of London, Series B. Biological Sciences, 204(1156):301-328, 1979. (30pgs). [cited by applicant]
Martin et al.,“A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” In Proceedings of the IEEE International Conference on Computer… [cited by applicant]
Mittal et al.,“Making a “Making a Completely Blind” Image Quality Analyzer,” IEEE Signal processing letters, 20(3):209-212, 2012. (4pgs). [cited by applicant]
Miyato et al.,“Spectral normalization for generative adversarial networks,” International Conference on Learning Representations (ICLR), 2018 (26pgs). [cited by applicant]
R.M. Neal, “Mcmc using hamiltonian dynamics,” Handbook of Markov Chain Monte Carlo, 2(11):2, 2011. (50pgs). [cited by applicant]
Nijkamp et al.,“Learning non-convergent non-persistent short-runMCMC toward energy-based model,” In Advances in Neural Information Processing Systems (NeurIPS), 2019.(11pgs). [cited by applicant]
Rott Shaham et al.,“SinGAN: Learning a Generative Model from a Single Natural Image,” In Proc. of the IEEE Conf. on Computer Vision & Pattern Recognition (CVPR), 2019. (11pgs). [cited by applicant]
Shocher et al.,“InGAN: Capturing and Retargeting the “DNA” of a Natural Image,” In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019. (10 pgs). [cited by applicant]
Barbu et al.,“Monte Carlo Methods,” Springer, 2020. (14pgs). [cited by applicant]
Blau et al.,“The Perception-Distortion Tradeoff” arXiv preprint arXiv: 1711.06077, 2020. (18pgs). [cited by applicant]
Clevert et al.,“Fast and accurate deep network learning by exponential linear units (ELUs),” arXiv preprint arXiv:1511.07289, 2016. (14pgs). [cited by applicant]
Du et al.,“Energy-based models for atomic-resolution protein conformations,” arXiv preprint arXiv:2004.13167, 2020. (16pgs). [cited by applicant]
Du et al.,“Implicit generation and modeling with energy based models,” In Advances in Neural Information Processing Systems (NeurIPS), 2019. (11pgs). [cited by applicant]
Gao et al.,“Learning generative convnets via multi-grid modeling and sampling,” Proceedings of the IEEE Conference on Computer Vision & Pattern Recognition (CVPR), 2018. (10p). [cited by applicant]
Gao et al.,“Flow contrastive estimation of energy-based models,” arXiv preprint arXiv:1912.00589, 2020. (13pgs). [cited by applicant]
Gao et al.,“Learning energy-based models by diffusion recovery likelihood,” arXiv preprint arXiv:2012.08125, 2021. (25pgs). [cited by applicant]
Gatys et al.,“A neural algorithm of artistic style,” arXiv preprint arXiv:1508.06576, 2015. (16pgs). [cited by applicant]
Geman et al.,“Stochastic relaxation, gibbs distributions, & the bayesian restoration of images,” IEEE Transactions on Pattern Analysis & Machine Intelligence (TPAMI), 1984.(21p). [cited by applicant]
Shocher et al.,““Zero-shot” super-resolution using deep internal learning,” In Proc. of the IEEE Conference on Computer Vision & Pattern Recognition (CVPR), 2018. (9 pgs). [cited by applicant]
Szegedy et al.,“Going deeper with convolutions,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. (9 pgs). [cited by applicant]
Tsai et al.,“Deep Image Harmonization,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. (9 pgs). [cited by applicant]
Ulyanov et al.,“Deep Image Prior,” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. (9 pgs). [cited by applicant]
Wu et al.,“Sparse and deep generalizations of the FRAME model,” Annals of Mathematical Sciences and Applications, 3(1):211-254, 2018. (44pgs). [cited by applicant]
Xie et al.,“Cooperative learning of energy-based model and latent variable model via memc teaching,” In Proc. of the AAAI Conf. on Artificial Intelligence (AAAI), 2018. (10pgs). [cited by applicant]
Xie et al.,“A theory of generative convnet,” In International Conference on Machine Learning (ICML), 2016. (10pgs). [cited by applicant]
Xie et al.,“Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification,” In Proc. of the IEEE Conf. on Computer Vision & Pattern Recognition (CVPR), 2021… [cited by applicant]
Xie et al.,“Learning Descriptor Networks for 3D Shape Synthesis and Analysis,” arXiv preprint arXiv:1804.00586, 2018. (10pgs). [cited by applicant]