Granted Patent B2

US 12,749,150 · App. 17/406,902 · Granted Sep 29, 2026

Inventors: Shiqiu Liu (Santa Clara, CA); Robert Pottorff (Santa Clara, CA); Guilin Liu (San Jose, CA); Karan Sapra (Santa Clara, CA); Jon Barker (Boulder, CO); David Tarjan (Mountain View, CA); Pekka Janis (Uusimaa, FI); Edvard Fagerholm (Uusimaa, FI); Lei Yang (Santa Clara, CA); Kevin Jonathan Shih (Santa Clara, CA); Marco Salvi (Seattle, WA); Timo Roman (Uusimaa, FI); Andrew Tao (Los Altos, CA); Bryan Catanzaro (Los Altos Hills, CA)

Assignee: NVIDIA Corporation

G06T3/4069G06T1/20G06T5/20G06T5/50G06T5/70H04N23/80H04N23/951G06T2207/10016G06T2207/20084

Abstract

Claims (35)

1 . A method, comprising:

obtaining lower resolution (LR) images;

generating, with a first neural network, one or more neural network parameters based, at least in part, on image data associated with the LR images at a first frame rate;

providing the one or more neural network parameters to a second neural network different from the first neural network; and

generating, with the second neural network at a different second frame rate, high resolution (HR) images using a jitter-aware upsampling process based, at least in part, on the LR images.

2 . The method of claim 1 , wherein the one or more neural network parameters are generated separately from generating the HR images with the second neural network.

3 . The method of claim 2 , wherein at least two HR images are generated at the different second frame rate that is different than the first frame rate associated with one or more frames of a sequence that are used to generate the one or more neural network parameters.

4 . The method of claim 1 , further comprising:

obtaining a video sequence of the LR images, and wherein the one or more neural network parameters are updated at a predetermined image interval along the video sequence.

5 . The method of claim 1 , wherein at least one of the one or more neural network parameters are generated to be used for a target image and enabled to be used on one or more images near the target image along a video sequence of the images.

6 . The method of claim 1 , further comprising:

inputting image data based, at least in part, on one or more of the LR images into a neural network to generate the one or more neural network parameters.

7 . The method of claim 6 , wherein the image data is at least associated with one or more sample locations in the LR images.

8 . The method of claim 1 , wherein at least one of the one or more neural network parameters is generated using the image data to generate weights.

9 . The method of claim 1 , wherein the jitter-aware upsampling process comprises applying one or more sub-pixel offsets to the LR images.

10 . A system for image processing, comprising:

at least one processor; and

at least one memory communicatively coupled to the at least one processor and storing a video sequence of low resolution (LR) images, the at least one processor being configured to operate by:

generating higher resolution (HR) images comprising inputting image data of the LR images into a second neural network;

generating, with a first neural network different from the second neural network, one or more neural network parameters based, at least in part on image data associated with the LR images at a first frame rate; and

providing the one or more neural network parameters to the second neural network to be used to generate the HR images using a jitter-aware upsampling process at a second different frame rate.

11 . The system of claim 10 , wherein the image data is generated in association with at least one neural network.

12 . The system of claim 10 , wherein the image data comprises pixel values representing at least part of an image.

13 . The system of claim 10 , wherein the generating the one or more neural network parameters comprises inputting the image data into at least one neural network.

14 . The system of claim 10 , wherein the second neural network uses at least one of available one or more neural network parameters rather than waiting for other one or more neural network parameters to be generated.

15 . A non-transitory computer-readable medium comprising instructions that, when performed by at least one processor of a computing device, cause the computing device to at least:

cause a first neural network to generate one or more neural network parameters based on a video sequence of low resolution (LR) images at a first frame rate;

provide the one or more neural network parameters to a second neural network different from the first neural network, the second neural network to perform high resolution (HR) image generation; and

cause the second neural network to generate HR images using a jitter-aware upsampling process, based on the LR images, at a second different frame rate.

16 . The non-transitory computer-readable medium of claim 15 , wherein the instructions that, when performed by the at least one processor of the computing device, cause the computing device to further cause the second neural network to generate HR images, based on the LR images, at the different second frame rate that is different than the first frame rate associated with the video sequence.

17 . The non-transitory computer-readable medium of claim 15 , wherein the instructions that, when performed by the at least one processor of the computing device, cause the computing device to generate the one or more neural network parameters separately from generating the HR images with the second neural network.

18 . The non-transitory computer-readable medium of claim 15 , wherein the first neural network comprises a convolutional neural network.

19 . The non-transitory computer-readable medium of claim 15 , wherein the one or more neural network parameters comprise one or more weights.

20 . The non-transitory computer-readable medium of claim 15 , wherein the instructions that, when performed by the at least one processor of the computing device, cause the computing device to train the second neural network on training images.

21 . The non-transitory computer-readable medium of claim 15 , wherein the instructions that, when performed by the at least one processor of the computing device, cause the computing device to input image data based, at least in part, on one or more the LR images into a neural network to generate the one or more neural network parameters.

Assignments (1)

Continuity (3)

References Cited (301)