IP Library › Granted Patent US 12,406,338
Granted Patent B2
US 12,406,338 · App. 18/169,545 · Granted Sep 2, 2025

Pseudoinverse guidance for data restoration with diffusion models

Inventor: Jiaming Song (San Carlos, CA)
Assignee: NVIDIA Corporation
G06T5/70G06T5/50G06T5/77G06T2207/20081G06T2207/20084G06T2207/20212
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,406,338
App. No.
18/169,545
Granted
Sep 2, 2025
Kind
B2
Abstract

A diffusion model is augmented with pseudoinverse guidance to restore data, removing artifacts and generating high-quality reconstructed data from limited, low-quality and/or noisy input data. The low-quality input data is denoised by a diffusion model and the denoised input data is combined with a guidance term to produce output data of higher-quality compared with the low-quality input data. The guidance term is a vector-Jacobian product that encourages consistency between the denoised input data and measurements after a pseudoinverse transformation. The denoising process may be applied in an iterative fashion to generate valid solutions to the inverse problem. The augmented diffusion model is a problem-agnostic (e.g., plug-and-play) denoiser that can restore data for a variety of tasks. Example image restoration tasks include denoising, JPEG denoising, deblurring, outpainting, inpainting, colorization, high-dynamic range, and super-resolution.

Claims (38)

1. A computer-implemented method, comprising:

receiving denoised data generated by a neural network model;

applying a function to the denoised data to compute degraded data, wherein the function is a data corruption process;

receiving noisy input data;

computing a result based on differences between the noisy input data and the degraded data after applying a pseudoinverse transformation of the function to each of the noisy input data and the degraded data; and

combining the result and the denoised data to produce reconstructed data with a reduced number of artifacts compared with the noisy input data.

2. The computer-implemented method of claim 1 , wherein the artifacts in the noisy input data result from applying the function to input data.

3. The computer-implemented method of claim 1 , wherein the function is JPEG encoding and the pseudoinverse transformation of the function is JPEG decoding.

4. The computer-implemented method of claim 1 , wherein the function is resolution reduction and the pseudoinverse transformation of the function is super resolution.

5. The computer-implemented method of claim 1 , wherein the function is masking and the pseudoinverse transformation of the function is inpainting.

6. The computer-implemented method of claim 1 , wherein the neural network model is pre-trained as a problem-agnostic denoising neural network model.

7. The computer-implemented method of claim 1 , wherein the noisy input data is acquired from a sensor.

8. The computer-implemented method of claim 7 , wherein the sensor captures an image.

9. The computer-implemented method of claim 1 , wherein the neural network model processes Gaussian noise to generate the denoised data.

10. The computer-implemented method of claim 1 , wherein the neural network model processes previous reconstructed data that approximates the noisy input data.

11. The computer-implemented method of claim 1 , wherein the result comprises a vector-Jacobian product.

12. The computer-implemented method of claim 1 , wherein computing the result comprises scaling a vector of the differences by a partial derivative of the denoised data with respect to previous reconstructed data with fewer artifacts compared with the noisy input data.

13. The computer-implemented method of claim 1 , wherein combining the result and the denoised data comprises accumulating the denoised data and the result.

14. The computer-implemented method of claim 1 , wherein the function is non-differentiable.

15. The computer-implemented method of claim 1 , wherein at least one of the steps of applying, computing, and combining is performed on a server or in a data center to generate the reconstructed data, and the reconstructed data is streamed to a user device.

16. The computer-implemented method of claim 1 , wherein at least one of the steps of applying, computing, and combining is performed within a cloud computing environment.

17. The computer-implemented method of claim 1 , wherein at least one of the steps of applying, computing, and combining is performed for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle.

18. The computer-implemented method of claim 1 , wherein at least one of the steps of applying, computing, and combining is performed on a virtual machine comprising a portion of a graphics processing unit.

19. A system, comprising:

a processor that is connected to the memory, wherein the processor is configured to reconstruct data by:

receiving denoised data generated by a neural network model;

applying a function to the denoised data to compute degraded data, wherein the function is a data corruption process;

receiving noisy input data;

computing a result based on differences between the noisy input data and the degraded data after applying a pseudoinverse transformation of the function to each of the noisy input data and the degraded data; and

combining the result and the denoised data to produce reconstructed data with a reduced number of artifacts compared with the noisy input data.

20. The system of claim 19 , wherein the artifacts in the noisy input data result from applying the function to input data.

21. A non-transitory computer-readable media storing computer instructions for data reconstruction that, when executed by one or more processors, cause the one or more processors to perform the steps of:

receiving denoised data generated by a neural network model;

applying a function to the denoised data to compute degraded data, wherein the function is a data corruption process;

receiving noisy input data;

computing a result based on differences between the noisy input data and the degraded data after applying a pseudoinverse transformation of the function to each of the noisy input data and the degraded data; and

combining the result and the denoised data to produce reconstructed data with a reduced number of artifacts compared with the noisy input data.

22. The non-transitory computer-readable media of claim 21 , wherein computing the result comprises scaling a vector of the differences by a partial derivative of the denoised data with respect to previous reconstructed data with fewer artifacts compared with the noisy input data.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 15, 2023
From: HYUN, CHANG HOON; CHO, JAE HONG
To: SPIGEN KOREA CO., LTD.
Reel/Frame 062709/0805 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 15, 2023
From: SONG, JIAMING
To: CORPORATION, NVIDIA
Reel/Frame 062710/0022 →
Continuity (2)
Provisional Application 63394776 · Aug 3, 2022
Related Publication 20240046422A1 · Feb 8, 2024
References Cited (76)
US 11847761B2 · Zhou · 2023 [cited by examiner]
US 20200003858A1 · Takeshima · 2020 [cited by examiner]
US 20200104745A1 · Li · 2020 [cited by examiner]
US 20200107378A1 · Velev · 2020 [cited by examiner]
US 20200311490A1 · Lee · 2020 [cited by examiner]
US 20230058096A1 · Ferrés · 2023 [cited by examiner]
US 20230103638A1 · Saharia · 2023 [cited by examiner]
US 20230236271A1 · Fessler · 2023 [cited by examiner]
CN 105164725A · 2015 [cited by examiner]
CN 113811921A · 2021 [cited by examiner]
JP 2019211475A · 2019 [cited by examiner]
Sohl-Dickstein, J., et al., “Deep unsupervised learning using nonequilibrium thermodynamics,” arXiv preprint arXiv:1503.03585, Mar. 2015. [cited by applicant]
Vahdat, A., et al., “Score-based generative modeling in latent space,” arXiv preprint arXiv:2106.05931, Jun. 2021. [cited by applicant]
Vincent, P., “A connection between score matching and denoising autoencoders,” Neural computation, 23(7):1661-1674, Jul. 2011. [cited by applicant]
Whang, J., et al., “Deblurring via stochastic refinement,” arXiv preprint arXiv:2112.02475, Dec. 2021. [cited by applicant]
Yu, F., et al., “LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop,” arXiv preprint arXiv:1506.03365, Jun. 2015. [cited by applicant]
Zhang, Q., et al., “Fast sampling of diffusion models with exponential integrator,” arXiv preprint arXiv:2204.13902, Apr. 2022. [cited by applicant]
Zhang, Q., et al., “gDDIM: Generalized denoising diffusion implicit models,” arXiv prepring arXiv:2206.05564, Jun. 2022. [cited by applicant]
Bora, A., et al., “Compressed sensing using generative models,” arXiv preprint arXiv:1703.03208, Mar. 2017. [cited by applicant]
Choi, J., et al., “ILVR: Conditioning method for denoising diffusion probabilistic models,” arXiv preprint arXiv:2108.02938, Aug. 2021. [cited by applicant]
Chung, H., et al., “Come-Closer-Diffuse-Faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction,” arXiv preprint arXiv:2112.05146, Dec. 2021. [cited by applicant]
Daras, G., et al., “Intermediate layer optimization for inverse problems using deep generative models,” arXiv preprint arXiv:2012.07364, Feb. 2021. [cited by applicant]
Dhariwal, P., et al., “Diffusion models beat GANs on image synthesis,” arXiv preprint arXiv:2105.05233, May 2021. [cited by applicant]
Ehrlich, M., et al., “Quantization guided JPEG artifact correction,” arXiv preprint arXiv:2004.09320, Apr. 2020. [cited by applicant]
He, K., et al., “Deep residual learning for image recognition,” arXiv preprint arXiv:1512.03385, Dec. 2015. [cited by applicant]
Heusel, M., et al., “GANs trained by a two time-scale update rule converge to a local nash equilibrium,” arXiv preprint arXiv:1706.08500, Jun. 2017. [cited by applicant]
Ho, J., et al., “Denoising diffusion probabilistic models,” arXiv preprint arXiv:2006.11239, Jun. 2020. [cited by applicant]
Jalal, A.,, et al., “Robust compressed sensing MRI with deep generative priors,” arXiv preprint arXiv:2108.01368, Aug. 2021. [cited by applicant]
Menon, S., et al., “PULSE: Self-Supervised photo unsampling via latent space exploration of generative models,” arXiv preprint arXiv:2003.03808, Mar. 2020. [cited by applicant]
Pan, X., et al., “Exploiting deep generative prior for versatile image restoration and manipulation,” IEEE transactions on pattern analysis and machine intelligence, PP, Sep. 2021. [cited by applicant]
Romano, Y., et al., “The little engine that could: Regularization by denoising (RED),” arXiv preprint arXiv:1611.02862, Nov. 2016. [cited by applicant]
Russakovsky, O., et al., “ImageNet large scale visual recognition challenge,” International Journal of Computer Vision, 115(3):211-252, Dec. 2015. [cited by applicant]
Santurkar, S., et al., “Image synthesis with a single (robust) classifier,” arXiv preprint arXiv:1906.09453, Jun. 2019. [cited by applicant]
Anderson, B.D., “Reverse-time diffusion equation models,” Stochastic processes and their applications, 12(3):313-326, May 1980. [cited by applicant]
Bardsley, J., “MCMC-based image reconstruction with uncertainty quantification,” Siam Journal on Scientific Computing, 34(3):A 1316-A 1332, Jun. 2012. [cited by applicant]
Binkowski, M., et al., “Demystifying MMD GANs,” arXiv preprint arXiv:1801.01401, Jan. 2021. [cited by applicant]
Chung, H., et al., “Score-based diffusion models for accelerated MRI,” Medical Image Analysis, pp. 102479, Jul. 2022. [cited by applicant]
Chung, H., et al., “Diffusion posterior sampling for general noisy inverse problems,” arXiv preprint arXiv:2209.14687, Nov. 2022. [cited by applicant]
Chung, H., et al., “Improving diffusion models for inverse problems using manifold constraints,” arXiv preprint arXiv:2206.00941, Oct. 2022. [cited by applicant]
Daras, G., et al., “Score-guided intermediate layer optimization: Fast langevin mixing for inverse problems,” arXiv preprint arXiv:2206.09104, Jun. 2022. [cited by applicant]
Daras, G., et al., “Soft diffusion: Score matching for general corruptions,” arXiv preprint arXiv:2209.05442, Oct. 2022. [cited by applicant]
Efron, B., “Tweedie's formula and selection bias,” Journal of American Statistical Association, 106(496):1602-1614, Apr. 2012. [cited by applicant]
Goodfellow, I., et al., “Generative adversarial networks,” In Advances in neural information processing systems, pp. 2672-2680, Jun. 2014. [cited by applicant]
Grenander, U., et al., “Representations of knowledge in complex systems,” Journal of the Royal Statistical Society: Series B (Methodological), 56(4):549-581, Nov. 1994. [cited by applicant]
Ho, J., et al., “Classifier-free diffusion guidance,” arXiv Preprint arXiv:2207.12598, Jul. 2022. [cited by applicant]
Ho, J., et al., “Video diffusion models,” arXiv preprint arXiv:2204.03458, Jun. 2022. [cited by applicant]
Hoogeboom, E., et al., “Blurring diffusion models,” arXiv preprint arXiv:2209.05557, Sep. 2022. [cited by applicant]
Jing, B., et al., “Subspace diffusion generative models,” arXiv preprint arXiv:2205.01490, May 2022. [cited by applicant]
Kadkhodaie, Z., et al., “Stochastic solutions for linear inverse problems using prior implicit in a denoiser,” Advances in Neural Information Processing Systems, 34:13242-13254, Jul. 2021. [cited by applicant]
Karras, T., et al., “Elucidating the design space of diffusion-based generative models,” arXiv preprint arXiv:2206.00364, Oct. 2022. [cited by applicant]
Kawar, B., et al., “Denoising diffusion restoration models,” arXiv preprint arXiv:2201.11793, Oct. 2022. [cited by applicant]
Kawar, B., et al., “JPEG artifact correction using denoising diffusion restoration models,” arXiv preprint arXiv:2209.11888, Nov. 2022. [cited by applicant]
Kong, Z., et al., “DiffWave: A versatile diffusion model for audio synthesis,” arXiv preprint arXiv:2009.09761, Mar. 2021. [cited by applicant]
Laumont, R., et al., “Bayesian imaging using plug and play priors: when langevin meets tweedie,” SIAM Journal on Imaging Sciences, 15(2):701-737, Jan. 2022. [cited by applicant]
Li, X.L., et al., “Diffusion-LM improves controllable text generation,” arXiv preprint arXiv:2205.14217, May 2022. [cited by applicant]
Liu, Y., et al., “Single-image HDR reconstruction by learning to reverse the camera pipeline,” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1651-1660, Apr. 2020. [cited by applicant]
Lu, C., et al., “DPM-solver: A fast ODE solver for diffusion probablistic model sampling in around 10 steps,” arXiv preprint arXiv:2206.00927, Oct. 2022. [cited by applicant]
Meng, C., et al., “SDEdit: Image synthesis and editing with stochastic differential equations,” arXiv preprint arXiv:2108.01073, Jan. 2022. [cited by applicant]
Ongie, G., et al., “Deep learning techniques for inverse problems in imaging,” IEEE Journal on Selected Areas in Information Theory, 1(1):39-56, May 2020. [cited by applicant]
Rissanen, S., et al., “Generative modelling with inverse heat dissiptation,” arXiv preprint arXiv:2206.13397, Dec. 2022. [cited by applicant]
Rombach, R., et al., “High-resolution image synthesis with latent diffusion models,” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, Apr. 2022. [cited by applicant]
Ryu, D., et al., “Pyramidal denoising diffusion probabilistic models,” arXiv preprint arXiv:2208.01864, Sep. 2022. [cited by applicant]
Saharia, C., et al., “Image super-resolution via iterative refinement,” arXiv preprint arXiv:2104.07636, Jun. 2021. [cited by applicant]
Saharia, C., et al., “Palette: Image-to-image diffusion models,” in ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, May 2022. [cited by applicant]
Saharia, C., et al., “Photorealistic text-to-image diffusion models with deep language understanding,” arXiv preprint arXiv:2205.11487, May 2022. [cited by applicant]
Saremi, S., et al., “Neural empirical bayes,” Journal of Machine Learning Research: JMLR, 20(181):1-23, Apr. 2020. [cited by applicant]
Sijbers, J., et al., “Maximum likelihood estimation of signal amplitude and noise variance from mr data,” Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medici… [cited by applicant]
Sinha, A., et al., “D2C: Diffusion-decoding models for few-shot conditional generation,” Advances in Neural Information Processing Systems, 34:12533-12548, Jun. 2021. [cited by applicant]
Song, J., et al., “Denoising diffusion implicit models,” In International Conference on Learning Representations, Oct. 2022. [cited by applicant]
Song, Y., et al., “Solving inverse problems in medical imaging with score-based generative models,” arXiv preprint arXiv:2111.08005, Jun. 2022. [cited by applicant]
Song, Y., et al., “Score-based generative modeling through stochastic differential equations,” In International Conference on Learning Representations, Feb. 2021. [cited by applicant]
Stein, C., “Estimation of the mean of a multivariate normal distribution,” The Annals of Statistics, vol. 9, No. 6, pp. 1135-1151, Nov. 1981. [cited by applicant]
Tashiro, Y., et al., “CSDI: Conditional score-based diffusion models for probabilistic time series imputation,” Advances in Neural Information Processing Systems, 34:24804-24816, Oct. 2021. [cited by applicant]
Ulyanov, D., et al., “Deep image prior,” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446-9454, May 2020. [cited by applicant]
Venkatakrishnan, S., et al., “Plug-and-play priors for model based reconstruction,” In 2013 IEEE Global Conference on Signal and Information Processing, pp. 945-948, IEEE, Dec. 2013. [cited by applicant]
Yu, J., et al., “Free-form image inpainting with gated convolution,” In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4471-4480, Oct. 2019. [cited by applicant]