IP Library Granted Patent US 12,373,916
Granted Patent B2
US 12,373,916 · App. 17/406,922 · Granted Jul 29, 2025

Image enhancement using one or more neural networks

Inventors: Robert Pottorff (Santa Clara, CA); David Tarjan (Mountain View, CA); Andrew Tao (Los Altos, CA); Bryan Catanzaro (Los Altos Hills, CA)
Assignee: NVIDIA CORPORATION
G06T3/4046G06T5/20G06T5/70G06T5/73G06T5/80G06T15/503
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,373,916
App. No.
17/406,922
Granted
Jul 29, 2025
Kind
B2
Abstract

Apparatuses, systems, and techniques are presented to generate images with one or more visual effects applied. In at least one embodiment, one or more visual effects are applied to one or more images having a resolution that is less than a first resolution and those visual effects approximated for one or more images having a resolution that is greater than or equal to the first resolution.

Claims (41)

1. A method of image processing comprising:

obtaining initial training images;

generating blurred training images comprising intentionally blurring one or more of the initial training images;

generating low resolution training images or high resolution training images or both and comprising using the blurred training images;

training a neural network to generate blurred high resolution pixel values comprising using the low resolution training images or high resolution training images or both; and

inputting lower resolution images to the neural network while separately generating weights or biases or both of the neural network based on statistics of the low resolution images.

2. The method of claim 1 , further comprising:

blurring the initial training images comprising changing image data of the initial training images.

3. The method of claim 1 , further comprising:

blurring the initial training images comprising applying a function-related value to image data of the initial training images.

4. The method of claim 1 , further comprising:

training the neural network to convert a resolution on input blurred lower resolution images.

5. The method of claim 1 , further comprising:

processing a blurred image to generate a desired size of a lower resolution training image or higher resolution training image or both.

6. At least one non-transitory article having at least one computer readable medium comprising a plurality of instructions that in response to being executed on a computing device, cause the computing device to operate by:

obtaining initial training images;

generating blurred training images comprising intentionally blurring one or more of the initial training images;

generating lower resolution training images or high resolution training images or both and comprising using the blurred training images;

training a neural network to generate blurred higher resolution pixel values comprising

using the lower resolution training images or higher resolution training images or both; and

inputting lower resolution images to the neural network while separately generating weights or biases or both of the neural network based on statistics of the low resolution images.

7. The at least one non-transitory article of claim 6 , wherein the instructions, in response to being executed, further cause the computing device to operate by generating a factor to compensate for reducing a size of the initial training images.

8. The at least one non-transitory article of claim 7 , wherein the factor is used to convolve the initial training image to form the blurred training images.

9. The at least one non-transitory article of claim 6 , wherein generating the blurred training images comprises intentionally blurring less than all of one or more of the initial training images.

10. A method, comprising:

generating one or more blurred images by, at least, blurring pixel values of one or more training images;

generating at least lower resolution training images or high resolution training images or both and comprising using the blurred images;

training a neural network to generate blurred images at a higher resolution using at least the lower resolution training images or higher resolution training images; and

inputting lower resolution images to the neural network while separately generating weights or biases or both of the neural network based on statistics of the low resolution images.

11. The method of claim 10 , further comprising: blurring the training images at least in part by changing image data of the training images.

12. The method of claim 10 , further comprising:

blurring the training images at least in part by applying a function-related value to image data of the training images.

13. The method of claim 10 , further comprising:

training the neural network to convert a resolution on input blurred lower resolution images.

14. The method of claim 10 , further comprising:

processing a blurred image to generate a desired size of at least one of a lower resolution training image or higher resolution training image.

15. The method of claim 10 , wherein a first parameterized function is determined for the one or more images having a resolution less than a first resolution, and wherein a second parameterized function is determined based on the first parameterized function and used to approximate one or more visual effects for the one or more training images having a resolution greater than or equal to the first resolution.

16. The method of claim 15 , further comprising:

applying one or more enhancements to the one or more training images, having a resolution greater than or equal to the first resolution, after one or more visual effects are approximated.

17. The method of claim 10 , further comprising:

downscaling the one or more training images, having less than a first resolution, to a resolution of an initial aliased training image generated by a rendering engine.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 28, 2022
From: POTTORFF, ROBERT; TARJAN, DAVID; TAO, ANDREW; CATANZARO, BRYAN
To: NVIDIA CORPORATION
Reel/Frame 062222/0931 →
Continuity (2)
Continuation 17012000 · Sep 3, 2020
Related Publication 20210383505A1 · Dec 9, 2021
References Cited (96)
US 5870505A · Wober et al. · 1999 [cited by applicant]
US 7149262B1 · Nayar · 2006 [cited by examiner]
US 10297045B2 · Sunkavalli · 2019 [cited by applicant]
US 10664963B1 · Rossi et al. · 2020 [cited by applicant]
US 10701394B1 · Caballero et al. · 2020 [cited by applicant]
US 10769764B2 · Fang et al. · 2020 [cited by applicant]
US 10817990B1 · Yang et al. · 2020 [cited by applicant]
US 11244195B2 · Pao et al. · 2022 [cited by applicant]
US 11328523B2 · Shectman et al. · 2022 [cited by applicant]
US 11423512B2 · Ozeki · 2022 [cited by applicant]
US 11449966B2 · Su et al. · 2022 [cited by applicant]
US 11521299B2 · Perazzi et al. · 2022 [cited by applicant]
US 11528492B2 · Wang et al. · 2022 [cited by applicant]
US 11528493B2 · Bae et al. · 2022 [cited by applicant]
US 11537849B2 · Zhu et al. · 2022 [cited by applicant]
US 11538136B2 · Yun et al. · 2022 [cited by applicant]
US 11651477B2 · Zhang et al. · 2023 [cited by applicant]
US 11727533B2 · Yoo · 2023 [cited by examiner]
US 11810268B2 · Pottorff et al. · 2023 [cited by applicant]
US 11836898B2 · He et al. · 2023 [cited by applicant]
US 11842267B2 · Liu et al. · 2023 [cited by applicant]
US 11880977B2 · Price et al. · 2024 [cited by applicant]
US 11948281B2 · Lin et al. · 2024 [cited by applicant]
US 11967046B2 · Stewart · 2024 [cited by applicant]
US 12190478B2 · Ozcan · 2025 [cited by examiner]
US 20020172434A1 · Freeman · 2002 [cited by examiner]
US 20040260662A1 · Staelin et al. · 2004 [cited by applicant]
US 20110115934A1 · Wang · 2011 [cited by examiner]
US 20110188770A1 · Weinberger · 2011 [cited by examiner]
US 20110211765A1 · Nagumo et al. · 2011 [cited by applicant]
US 20130034313A1 · Lin · 2013 [cited by examiner]
US 20140321738A1 · Shibata · 2014 [cited by examiner]
US 20150071545A1 · Yang · 2015 [cited by examiner]
US 20160055627A1 · Shibata · 2016 [cited by examiner]
US 20160180600A1 · Liu · 2016 [cited by applicant]
US 20180197278A1 · Lee et al. · 2018 [cited by applicant]
US 20180293706A1 · Viswanathan et al. · 2018 [cited by applicant]
US 20180300855A1 · Tang · 2018 [cited by examiner]
US 20180336662A1 · Kimura · 2018 [cited by examiner]
US 20190188535A1 · Chen et al. · 2019 [cited by applicant]
US 20190206026A1 · Vemulapalli et al. · 2019 [cited by applicant]
US 20190303720A1 · Karam · 2019 [cited by examiner]
US 20190306923A1 · Xiong et al. · 2019 [cited by applicant]
US 20190333199A1 · Ozcan · 2019 [cited by examiner]
US 20190370936A1 · Zhang et al. · 2019 [cited by applicant]
US 20190385288A1 · Stewart · 2019 [cited by applicant]
US 20200018944A1 · Fang · 2020 [cited by examiner]
US 20200068617A1 · Yoon et al. · 2020 [cited by applicant]
US 20200111004A1 · Wang · 2020 [cited by examiner]
US 20200126263A1 · Dinh et al. · 2020 [cited by applicant]
US 20200258204A1 · Fang et al. · 2020 [cited by applicant]
US 20200285883A1 · Hiasa · 2020 [cited by examiner]
US 20200364831A1 · Ozeki · 2020 [cited by applicant]
US 20200366938A1 · Makeev et al. · 2020 [cited by applicant]
US 20200372611A1 · Liu · 2020 [cited by examiner]
US 20200380639A1 · Rossi et al. · 2020 [cited by applicant]
US 20200389672A1 · Kennett et al. · 2020 [cited by applicant]
US 20210049741A1 · Yoo · 2021 [cited by examiner]
US 20210073945A1 · Kim · 2021 [cited by examiner]
US 20210073959A1 · Elmalem · 2021 [cited by examiner]
US 20210097646A1 · Choi et al. · 2021 [cited by applicant]
US 20210112261A1 · Hwang et al. · 2021 [cited by applicant]
US 20210150678A1 · Yi et al. · 2021 [cited by applicant]
US 20210166350A1 · Wang · 2021 [cited by examiner]
US 20210304357A1 · Bae et al. · 2021 [cited by applicant]
US 20210342974A1 · Zhang et al. · 2021 [cited by applicant]
US 20210342984A1 · Lin et al. · 2021 [cited by applicant]
US 20210352307A1 · Bae et al. · 2021 [cited by applicant]
US 20220148285A1 · Pao et al. · 2022 [cited by applicant]
US 20220327657A1 · Zheng et al. · 2022 [cited by applicant]
US 20220353435A1 · Cohen-Tidhar et al. · 2022 [cited by applicant]
US 20220366544A1 · Kudelski et al. · 2022 [cited by applicant]
US 20220375045A1 · Wu et al. · 2022 [cited by applicant]
US 20220405883A1 · Gou et al. · 2022 [cited by applicant]
US 20230019851A1 · Yi et al. · 2023 [cited by applicant]
US 20230138053A1 · Liu et al. · 2023 [cited by applicant]
CN 105120130A · 2015 [cited by applicant]
CN 109255755A · 2019 [cited by examiner]
CN 109345449A · 2019 [cited by examiner]
CN 109509160A · 2019 [cited by examiner]
CN 109544457A · 2019 [cited by applicant]
CN 109559278A · 2019 [cited by examiner]
CN 110335199A · 2019 [cited by examiner]
CN 110363704A · 2019 [cited by examiner]
CN 110674862A · 2020 [cited by examiner]
CN 111340711A · 2020 [cited by examiner]
EP 3828811A1 · 2021 [cited by applicant]
GB 2573486A · 2019 [cited by applicant]
WO WO2020015167A1 · 2020 [cited by examiner]
WO 2021231042A1 · 2021 [cited by applicant]
IEEE “IEEE Standard for Floating-Point Arithmetric”, Microprocessor Standards Committee of the IEEE Computer Society, IEEE Std 754-2008, dated Jun. 12, 2008, 70 pages. [cited by applicant]
United Kingdom Combined Search and Examination Report for Patent Application No. 2112524.0 dated Mar. 10, 2022, 6 pages. [cited by applicant]
United Kingdom Search and Exmaination Report for Application No. GB2112524.0, mailed Aug. 1, 2023, 4 pages. [cited by applicant]
Chinese Office Action for Appplication No. 202111027910.5 mailed May 12, 2025, 27 pages. [cited by applicant]
Gu et al., “Integration of Spatial-Spectral Information for Resolution Enhancement in Hyperspectral Images,” IEEE Transactions on Geoscience and Remote Sensing, 46(5): May 1, 2008, 12 pages. [cited by applicant]
Pengfei, et al., Image Super-resolution Reconstruction Based on Convolutional Neural Networks, pp. 197-202, Computer Engineering and Applications, vol. 55, No. 9, Jun. 22, 2018, 5 Pages. [cited by applicant]