IP Library › Granted Patent US 12,555,186
Granted Patent B2
US 12,555,186 · App. 17/172,330 · Granted Feb 17, 2026

Upsampling an image using one or more neural networks

Inventors: Shiqiu Liu (Santa Clara, CA); Robert Pottorff (Santa Clara, CA); Guilin Liu (San Jose, CA); Karan Sapra (Santa Clara, CA); Jon Barker (Boulder, CO); David Tarjan (Mountain View, CA); Pekka Janis (Uusimaa, FI); Edvard Fagerholm (Uusimaa, FI); Lei Yang (Santa Clara, CA); Kevin Shih (Santa Clara, CA); Marco Salvi (Seattle, WA); Timo Roman (Uusimaa, FI); Andrew Tao (Los Altos, CA); Bryan Catanzaro (Los Altos Hills, CA)
Assignee: NVIDIA Corporation
G06T3/4069G06T1/20G06T5/20G06T5/50G06T5/70H04N23/80H04N23/951G06T2207/10016G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,555,186
App. No.
17/172,330
Filed
Feb 10, 2021
Granted
Feb 17, 2026
Kind
B2
Examiner
TRAN, PHUOC
Art Unit
2668
USPC
382/299
Abstract

Apparatuses, systems, and techniques are presented to generate images. In at least one embodiment, one or more neural networks are used to generate one or more images using one or more pixel weights determined based, at least in part, on one or more sub-pixel offset values.

Claims (66)

1 . One or more processors, comprising:

circuitry to use one or more neural networks to;

determine a first blending weight and a second blending weight based, at least in part, on one or more upsampled images generated using one or more sub-pixel offsets associated with one or more low resolution images;

perform, using the first blending weight, a first blending of pixel values of the one or more upsampled images with pixel values of one or more prior upsampled images to generate one or more intermediate image representations;

perform, using the second blending weight, a second blending of the pixel values of the one or more upsampled images with pixel values of the one or more intermediate image representations; and

generate one or more output images.

2 . The one or more processors of claim 1 , wherein the one or more upsampled images are generated using a jitter-aware upsampling process.

3 . The one or more processors of claim 2 , wherein the jitter-aware sampling process comprises rendering an image sequence to include one or more color values for one or more pixels of the one or more low resolution images and applying jitter between at least two or more low resolution images of the one or more low resolution images in the image sequence.

4 . The one or more processors of claim 1 , wherein the first blending applies only to pixel values of the one or more upsampled images for pixels that correspond to sample locations as determined using the one or more sub-pixel offsets.

5 . The one or more processors of claim 3 , wherein applying jitter comprises shifting a center point of a color determination for the one or more color values to another point in the one or more pixels.

6 . The one or more processors of claim 1 , wherein the first blending weight is higher than the second blending weight.

7 . A method comprising:

using one or more neural networks to;

determine a first blending weight and a second blending weight based, at least in part, on one or more upsampled images generated using one or more sub-pixel offsets associated with one or more low resolution images;

perform, using the first blending weight, a first blending of pixel values of the one or more upsampled images with pixel values of one or more prior upsampled images to generate one or more intermediate image representations;

perform, using the second blending weight, a second blending of the pixel values of the one or more upsampled images with pixel values of the one or more intermediate image representations; and

generate one or more output images.

8 . The method of claim 7 ,

wherein the one or more sub-pixel offsets are to be determined from a jitter-aware upsampling process.

9 . The method of claim 8 , wherein the jitter-aware sampling process comprises rendering an image sequence to include one or more color values for one or more pixels of the one or more low resolution images and applying jitter between at least two or more low resolution images of the one or more low resolution images in the image sequence.

10 . The method of claim 7 , wherein the first blending applies only to pixel values of the one or more upsampled images for pixels that correspond to sample locations determined using the one or more sub-pixel offsets.

11 . The method of claim 9 , wherein applying jitter comprises shifting a center point of a color determination for the one or more color values to another point in the one or more pixels.

12 . The method of claim 7 , wherein the first blending weight is higher than the second blending weight.

13 . A server, comprising:

one or more processors to use one or more neural networks to;

determine a first blending weight and a second blending weight based, at least in part, on one or more upsampled images generated using one or more sub-pixel offsets associated with one or more low resolution images;

perform, using the first blending weight, a first blending of pixel values of the one or more upsampled images with pixel values of one or more prior upsampled images to generate one or more intermediate image representations;

perform, using the second blending weight, a second blending of the pixel values of the one or more upsampled images with pixel values of the one or more intermediate image representations; and

generate one or more output images.

14 . The server of claim 13 , wherein the one or more sub-pixel offsets are to be determined from a jitter-aware upsampling process.

15 . The server of claim 14 , wherein the jitter-aware sampling process comprises rendering an image sequence to include one or more color values for one or more pixels of the one or more low resolution images and applying jitter between at least two or more low resolution images of the one or more low resolution images in the image sequence.

16 . The server of claim 13 , wherein the first blending applies only to pixel values of the one or more upsampled images for pixels that correspond to sample locations as determined using the one or more sub-pixel offsets.

17 . The server of claim 15 , wherein applying jitter comprises shifting a center point of a color determination for the one or more color values to another point in the one or more pixels.

18 . The server of claim 13 , wherein the first blending weight is higher than the second blending weight.

19 . A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least use one or more neural networks to:

determine a first blending weight and a second blending weight based, at least in part, on one or more upsampled images generated using one or more sub-pixel offsets associated with one or more low resolution images;

perform, using the first blending weight, a first blending of pixel values of the one or more upsampled images with pixel values of one or more prior upsampled images to generate one or more intermediate image representations;

perform, using the second blending weight, a second blending of the pixel values of the one or more upsampled images with pixel values of the one or more intermediate image representations; and

generate one or more output images.

20 . The non-transitory machine-readable medium of claim 19 , wherein the one or more sub-pixel offsets are to be determined from a jitter-aware upsampling process.

21 . The non-transitory machine-readable medium of claim 20 , wherein the jitter-aware sampling process comprises rendering an image sequence to include one or more color values for one or more pixels of the one or more low resolution images and applying jitter between at least two or more low resolution images of the one or more low resolution images in the image sequence.

22 . The non-transitory machine-readable medium of claim 19 , wherein the first blending applies only to pixel values of the one or more upsampled images for pixels that correspond to sample locations as determined using the one or more sub-pixel offsets.

23 . The non-transitory machine-readable medium of claim 21 , wherein applying jitter comprises shifting a center point of a color determination for the one or more color values to another point in the one or more pixels.

24 . The non-transitory machine-readable medium of claim 19 , wherein the first blending weight is higher than the second blending weight.

25 . A video game system, comprising:

one or more processors to use one or more neural networks to;

determine a first blending weight and a second blending weight based, at least in part, on one or more upsampled images generated using one or more sub-pixel offsets associated with one or more low resolution images;

perform, using the first blending weight, a first blending of pixel values of the one or more upsampled images with pixel values of one or more prior upsampled images to generate one or more intermediate image representations;

perform, using the second blending weight, a second blending of the pixel values of the one or more upsampled images with pixel values of the one or more intermediate image representations; and

generate one or more output images.

26 . The video game system of claim 25 , wherein the one or more sub-pixel offsets are to be determined from a jitter-aware upsampling process.

27 . The video game system of claim 26 , wherein the jitter-aware sampling process comprises rendering an image sequence to include one or more color values for one or more pixels of the one or more low resolution images and applying jitter between at least two or more low resolution images of the one or more low resolution images in the image sequence.

28 . The video game system of claim 25 , wherein the first blending applies only to pixel values of the one or more upsampled images for pixels that correspond to sample locations as determined using the one or more sub-pixel offsets.

29 . The video game system of claim 27 , wherein applying jitter comprises shifting a center point of a color determination for the one or more color values to another point in the one or more pixels.

30 . The video game system of claim 25 , wherein the first blending weight is higher than the second blending weight.

31 . An autonomous vehicle, comprising:

one or more processors to use one or more neural networks to;

determine a first blending weight and a second blending weight based, at least in part, on one or more upsampled images generated using one or more sub-pixel offsets associated with one or more low resolution images;

perform, using the first blending weight, a first blending of pixel values of the one or more upsampled images with pixel values of one or more prior upsampled images to generate one or more intermediate image representations;

perform, using the second blending weight, a second blending of the pixel values of the one or more upsampled images with pixel values of the one or more intermediate image representations; and

generate one or more output images.

32 . The autonomous vehicle of claim 31 , wherein the one or more sub-pixel offsets are to be determined from a jitter-aware upsampling process.

33 . The autonomous vehicle of claim 32 , wherein the jitter-aware sampling process comprises rendering an image sequence to include one or more color values for one or more pixels of the one or more low resolution images and applying jitter between at least two or more low resolution images of the one or more low resolution images in the image sequence.

34 . The autonomous vehicle of claim 31 , wherein the first blending applies only to pixel values of the one or more upsampled images for pixels that correspond to sample locations as determined using the one or more sub-pixel offsets.

35 . The autonomous vehicle of claim 33 , wherein applying jitter comprises shifting a center point of a color determination for the one or more color values to another point in the one or more pixels.

36 . The autonomous vehicle of claim 31 , wherein one or more navigation instructions are determined based, at least in part, upon locations of one or more objects identified in the output images.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 17, 2022
From: LIU, SHIQIU; POTTORFF, ROBERT; LIU, GUILIN; SAPRA, KARAN; BARKER, JON; TARJAN, DAVID; JANIS, PEKKA; FAGERHOLM, EDVARD; YANG, LEI; SHIH, KEVIN; SALVI, MARCO; ROMAN, TIMO; TAO, ANDREW; CATANZARO, BRYAN
To: NVIDIA CORPORATION
Reel/Frame 060836/0812 →
Continuity (2)
Continuation 17066282 · Oct 8, 2020
Related Publication 20220114701A1 · Apr 14, 2022
References Cited (292)
US 6028608A · Jenkins · 2000 [cited by applicant]
US 6801655B2 · Woodall · 2004 [cited by applicant]
US 7346208B2 · Staelin et al. · 2008 [cited by applicant]
US 7554540B2 · Hayes · 2009 [cited by applicant]
US 8345962B2 · Yu et al. · 2013 [cited by applicant]
US 8913822B2 · Matsuda et al. · 2014 [cited by applicant]
US 8934542B2 · Wiegand et al. · 2015 [cited by applicant]
US 9430817B2 · Schelten et al. · 2016 [cited by applicant]
US 9785886B1 · Andoni et al. · 2017 [cited by applicant]
US 9915625B2 · Gao et al. · 2018 [cited by applicant]
US 10430913B2 · Chen et al. · 2019 [cited by applicant]
US 10482584B1 · Kim et al. · 2019 [cited by applicant]
US 10552739B1 · Rausch et al. · 2020 [cited by applicant]
US 10701394B1 · Caballero et al. · 2020 [cited by applicant]
US 11256990B2 · Lanctot et al. · 2022 [cited by applicant]
US 11270603B1 · Bansal et al. · 2022 [cited by applicant]
US 20060159369A1 · Young · 2006 [cited by examiner]
US 20060233258A1 · Holcomb · 2006 [cited by applicant]
US 20080232452A1 · Sullivan et al. · 2008 [cited by applicant]
US 20110141348A1 · Jia et al. · 2011 [cited by applicant]
US 20120051426A1 · Takagi et al. · 2012 [cited by applicant]
US 20120219236A1 · Ali et al. · 2012 [cited by applicant]
US 20140204111A1 · Vaidyanathan et al. · 2014 [cited by applicant]
US 20140328400A1 · Puri et al. · 2014 [cited by applicant]
US 20140333669A1 · Stich · 2014 [cited by applicant]
US 20160117800A1 · Korkin · 2016 [cited by examiner]
US 20160353164A1 · Smolic et al. · 2016 [cited by applicant]
US 20160378863A1 · Shlens et al. · 2016 [cited by applicant]
US 20170091619A1 · Towal et al. · 2017 [cited by applicant]
US 20170124742A1 · Hasselgren et al. · 2017 [cited by applicant]
US 20170169602A1 · Blackmon et al. · 2017 [cited by applicant]
US 20170206405A1 · Molchanov et al. · 2017 [cited by applicant]
US 20170235476A1 · Hughes et al. · 2017 [cited by applicant]
US 20170256033A1 · Tuzel et al. · 2017 [cited by applicant]
US 20170272722A1 · Salvi et al. · 2017 [cited by applicant]
US 20170308734A1 · Chalom et al. · 2017 [cited by applicant]
US 20170345140A1 · Zhang et al. · 2017 [cited by applicant]
US 20180007362A1 · Krishnan · 2018 [cited by applicant]
US 20180025257A1 · van den Oord et al. · 2018 [cited by applicant]
US 20180075343A1 · van den Oord et al. · 2018 [cited by applicant]
US 20180129918A1 · Wang et al. · 2018 [cited by applicant]
US 20180253645A1 · Burr · 2018 [cited by applicant]
US 20180293707A1 · El-Khamy et al. · 2018 [cited by applicant]
US 20180293713A1 · Vogels et al. · 2018 [cited by applicant]
US 20180295320A1 · Breternitz et al. · 2018 [cited by applicant]
US 20180311663A1 · Giera et al. · 2018 [cited by applicant]
US 20180314938A1 · Andoni et al. · 2018 [cited by applicant]
US 20180357537A1 · Munkberg et al. · 2018 [cited by applicant]
US 20190014320A1 · Navarrete Michelini et al. · 2019 [cited by applicant]
US 20190035363A1 · Schluessler et al. · 2019 [cited by applicant]
US 20190043242A1 · Risser · 2019 [cited by examiner]
US 20190073591A1 · Andoni et al. · 2019 [cited by applicant]
US 20190096032A1 · Li · 2019 [cited by applicant]
US 20190096046A1 · Kalantari et al. · 2019 [cited by applicant]
US 20190108618A1 · Hwang et al. · 2019 [cited by applicant]
US 20190180418A1 · Kuybeda · 2019 [cited by applicant]
US 20190205736A1 · Bleiweiss et al. · 2019 [cited by applicant]
US 20190206026A1 · Vemulapalli et al. · 2019 [cited by applicant]
US 20190251397A1 · Tremblay et al. · 2019 [cited by applicant]
US 20190251401A1 · Shechtman et al. · 2019 [cited by applicant]
US 20190279076A1 · Hu et al. · 2019 [cited by applicant]
US 20190304067A1 · Vogels · 2019 [cited by examiner]
US 20190342606A1 · Kurokawa · 2019 [cited by applicant]
US 20190379883A1 · Wang et al. · 2019 [cited by applicant]
US 20200020082A1 · Zahneisen et al. · 2020 [cited by applicant]
US 20200027198A1 · Vogels et al. · 2020 [cited by applicant]
US 20200027202A1 · Oh et al. · 2020 [cited by applicant]
US 20200045348A1 · Boyce et al. · 2020 [cited by applicant]
US 20200050923A1 · Patney et al. · 2020 [cited by applicant]
US 20200077023A1 · Kang et al. · 2020 [cited by applicant]
US 20200089998A1 · Zagaynov et al. · 2020 [cited by applicant]
US 20200090305A1 · El-Khamy et al. · 2020 [cited by applicant]
US 20200126191A1 · Munkberg et al. · 2020 [cited by applicant]
US 20200134797A1 · Zhang · 2020 [cited by examiner]
US 20200160105A1 · Kehl · 2020 [cited by applicant]
US 20200186544A1 · Dichiu et al. · 2020 [cited by applicant]
US 20200226418A1 · Dorai-Raj et al. · 2020 [cited by applicant]
US 20200265567A1 · Hu et al. · 2020 [cited by applicant]
US 20200302236A1 · Gao et al. · 2020 [cited by applicant]
US 20200327702A1 · Wang et al. · 2020 [cited by applicant]
US 20200364509A1 · Weinzaepfel et al. · 2020 [cited by applicant]
US 20200364834A1 · Ferrés et al. · 2020 [cited by applicant]
US 20200402205A1 · Su et al. · 2020 [cited by applicant]
US 20210125583A1 · Kaplanyan et al. · 2021 [cited by applicant]
US 20210142176A1 · Ratner et al. · 2021 [cited by applicant]
US 20210142455A1 · Hsiao et al. · 2021 [cited by applicant]
US 20210142479A1 · Phogat et al. · 2021 [cited by applicant]
US 20210150674A1 · Cai et al. · 2021 [cited by applicant]
US 20210232897A1 · Bichler et al. · 2021 [cited by applicant]
US 20210233210A1 · Elron et al. · 2021 [cited by applicant]
US 20210272258A1 · Sharma et al. · 2021 [cited by applicant]
US 20210279950A1 · Phalak · 2021 [cited by applicant]
US 20210303912A1 · Yu et al. · 2021 [cited by applicant]
US 20210342670A1 · van den Oord et al. · 2021 [cited by applicant]
US 20210342974A1 · Zhang et al. · 2021 [cited by applicant]
US 20210366082A1 · Xiao · 2021 [cited by examiner]
US 20210383537A1 · Chitiboi et al. · 2021 [cited by applicant]
US 20220058420A1 · Peng et al. · 2022 [cited by applicant]
US 20220108425A1 · Chen et al. · 2022 [cited by applicant]
US 20220128724A1 · Sen et al. · 2022 [cited by applicant]
US 20220138903A1 · Liu et al. · 2022 [cited by applicant]
CN 103873915A · 2014 [cited by applicant]
CN 105408935A · 2016 [cited by applicant]
CN 106204449A · 2016 [cited by applicant]
CN 106331433A · 2017 [cited by applicant]
CN 108053456A · 2018 [cited by applicant]
CN 108805794A · 2018 [cited by applicant]
CN 108989731A · 2018 [cited by applicant]
CN 109034385A · 2018 [cited by applicant]
CN 109785235A · 2019 [cited by applicant]
CN 109862370A · 2019 [cited by applicant]
CN 109952577A · 2019 [cited by applicant]
CN 110046537A · 2019 [cited by applicant]
CN 110100252A · 2019 [cited by applicant]
CN 110458778A · 2019 [cited by applicant]
CN 110582748A · 2019 [cited by applicant]
CN 111402379A · 2020 [cited by applicant]
CN 111783817A · 2020 [cited by applicant]
CN 111798450A · 2020 [cited by applicant]
DE 69739577 · 2009 [cited by applicant]
DE 102018113845A1 · 2018 [cited by applicant]
DE 102021128623A1 · 2022 [cited by applicant]
EP 3739523A1 · 2020 [cited by applicant]
JP 2000041253A · 2000 [cited by applicant]
JP 2012515493A · 2012 [cited by applicant]
JP 2019132798A · 2019 [cited by applicant]
KR 20050100321A · 2005 [cited by applicant]
WO 0131568A1 · 2001 [cited by applicant]
WO 2012073865A1 · 2012 [cited by applicant]
WO 2015015195A1 · 2015 [cited by applicant]
WO 2017136784A1 · 2017 [cited by applicant]
WO 2018072308A1 · 2018 [cited by applicant]
WO 2019066794A1 · 2019 [cited by applicant]
WO 2019157599A1 · 2019 [cited by applicant]
WO 2019180414A1 · 2019 [cited by applicant]
WO 2020068140A1 · 2020 [cited by applicant]
WO 2020172043A2 · 2020 [cited by applicant]
WO 2020192471A1 · 2020 [cited by applicant]
Gu et al., “Integration of Spatial-Spectral Information for Resolution Enhancement in Hyperspectral Images,” IEEE Transactions on Geoscience and Remote Sensing, 46(5): May 1, 2008. [cited by applicant]
IEEE, “IEEE Standard 754-2008 (Revision of IEEE Standard 754-1985): IEEE Standard for Floating-Point Arithmetic,” Aug. 29, 2008, 70 pages. [cited by applicant]
International Search Report and Written Opinion for Application No. PCT/US2021/053803, mailed Feb. 3, 2022, filed Oct. 6, 2021, 11 pages. [cited by applicant]
Miravet et al., “A Two-Step Neural-Network Based Algorithm for Fast Image Super-Resolution,” Image and Vision Computing, Elsevier, Guildford, 25(9): Jun. 14, 2007. [cited by applicant]
Shi et al., Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network, IEEE Conference on Computer Vision and Pattern Recognition, Jun. 27, 2016, 10 pages. [cited by applicant]
Combined Search and Examination Report for United Kingdom Application No. GB2316422.1, mailed Nov. 22, 2023, 4 pages. [cited by applicant]
Notice of Allowance for Korean Application No. 10-2022-7011734, mailed Jan. 24, 2024, 6 pages. [cited by applicant]
Office Action for Korean Application No. 10-2022-7015877, mailed Jan. 25, 2024, 5 pages. [cited by applicant]
Office Action for Korean Application No. 10-2022-7015877, mailed Jul. 20, 2023, 12 pages. [cited by applicant]
Hayat: “Super-Resolution via Deep Learning,” Jun. 28, 2017, 33 pages. [cited by applicant]
Hsiao et al., “Video Enhancement via Super-Resolution Using Deep Quality Transfer Network,” 9 Big Data Analytics in the Social and Ubiquitous Context 5th International Workshop on Modeling Social Media, XP047407319, Mar… [cited by applicant]
International Search Report and Written Opinion for Application No. PCT/US2020/047898, dated Nov. 30, 2020, 11 pages. [cited by applicant]
Notice of Acceptance for Australian Application No. 2020346707, mailed Jul. 10, 2023, 3 pages. [cited by applicant]
Office Action for Australian Application No. 2020346707, mailed Jan. 16, 2023, 3 pages. [cited by applicant]
Office Action for Korean Patent Application No. 10-2022-7011734, mailed Jun. 29, 2023, 7 pages. [cited by applicant]
Office Action for United Kingdom Application No. GB2201094.6, mailed Mar. 27, 2023, 2 pages. [cited by applicant]
Saeckinger et al., “Application of the ANNA Neural Network Chip to High-Speed Character Recognition,” Mar. 18, 1991, 22 pages. [cited by applicant]
Salvi et al., “Surface Based Anti-Aliasing,” Proceedings of the ACM Siggraph Symposium on Interactive 3D Graphics and Games, 2012, 6 pages. [cited by applicant]
Salvi, “Deep Learning: The Future of Real-Time Rendering?” Nvidia, 2017, 53 pages. [cited by applicant]
Schied et al., “Spatiotemporal Variance-Guided Filtering: Real-Time Reconstruction for Path-Traced Global Illumination,” iIn Proceedings of HPG, 2017, 12 pages. [cited by applicant]
Schmidhuber, “Deep Learning in Neural Networks: An Overview,” Oct. 8, 2014, 88 pages. [cited by applicant]
Simonyan et al., “Very Deep Convolutional Networks for Large-Scale Image Recognition,” Apr. 10, 2015, 14 pages. [cited by applicant]
Son et al., “Fast Non-blind Deconvolution via Regularized Residual Networks with Long/Short Skip-Connections,” 2017, 10 pages. [cited by applicant]
Sze et al., Efficient Processing of Deep Neural Networks: A Tutorial and Survey, Aug. 13, 2017, 32 pages. [cited by applicant]
Tsafi, “Image Scaling using Deep Convolutional Neural Networks,” retrieved from https://engineering.flipboard.com/2015/05/scaling-convnets, Jun. 26, 2018, 36 pages. [cited by applicant]
Vincent et al., “Extracting and Composing Robust Features with Denoising Autoencoders,” Proceedings of the 25th International Conference on Machine Learning, 2008, 8 pages. [cited by applicant]
Wang et al., “Image Quality Assessment: From Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, 13(4): Apr. 2004, 14 pages. [cited by applicant]
Xie et al., “Image Denoising and Inpainting with Deep Neural Networks,” Proceedings of the 25th International Conference on Neural Information Processing Systems, 2012, 9 pages. [cited by applicant]
Yang et al., “Amortized Supersampling,” ACM Transactions on Graphics, 28(5): Dec. 2009, 12 pages. [cited by applicant]
Abadi et al., “TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems,” retrieved from https://www.tensorflow.org/, 2015, 10 pages. [cited by applicant]
Akeley, “Reality Engine Graphics,” Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques, 1993, 8 pages. [cited by applicant]
Bako et al., “Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings,” 2017, 14 pages. [cited by applicant]
Bavoil et al., “Aggregate G-Buffer Anti-Aliasing in Unreal Engine 4,” 2016, 45 pages. [cited by applicant]
Bengio et al., “Recurrent Neural Networks for Adaptive Temporal Processing,” AT&T Bell Laboratories, 1993, 35 pages. [cited by applicant]
Benty et al., “The Falcor Rendering Framework,” retrieved from https://github.com/NVIDIAGameWorks/Falcor, 2019, 4 pages. [cited by applicant]
Bigdeli et al., “Image Restoration using Autoencoding Priors,” CoRR, 2017, 12 pages. [cited by applicant]
Chaitanya et al., “Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder,” ACM Trans. Graph. 36, 4, Article 98, 2017, 12 pages. [cited by applicant]
Chajdas et al., “Subpixel Reconstruction Antialiasing for Deferred Shading,” Symposium on Interactive 3D Graphics and Games, 2011, 7 pages. [cited by applicant]
Chollet et al., “Keras,” GitHub, retrieved from https://github.com/fchollet/keras, 2015, 7 pages. [cited by applicant]
Clarberg et al., “A Sort-based Deferred Shading Architecture for Decoupled Sampling,” ACM Transactions on Graphics, 32(4): 2013, 10 pages. [cited by applicant]
Deering et al., “The Triangle Processor and Normal Vector Shader: A VLSI System for High Performance Graphics,” Proceedings of the 15th Annual Conference on Computer Graphics and Interactive Techniques, 22(4): Aug. 1988… [cited by applicant]
Dosovitskiy et al., “Generating Images with Perceptual Similarity Metrics Based on Deep Networks,” Feb. 9, 2016, 14 pages. [cited by applicant]
Durant et al., “Nvidia Tesla V100 GPU Architecture Whitepaper. Technical Report. Nvidia Corporation,” retrieved from http://www. nvidia.com/object/volta-architecture-whitepaper.html, 2017, 58 pages. [cited by applicant]
Epic Games, “Unreal Engine 4,” Wikipedia, 2012, 27 pages. [cited by applicant]
Fatahalian et al., “Reducing Shading on GPUs Using Quadfragment Merging,” ACM Siggraph, 2010, 8 pages. [cited by applicant]
Fedkiw et al., “Visual Simulation of Smoke,” Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, 2001, 8 pages. [cited by applicant]
Fuchs et al., “Fast Spheres, Shadows, Textures, Transparencies, and Image Enhancements in Pixel-Planes,” Advances in Computer Graphics, 1985, 10 pages. [cited by applicant]
Haeberli et al., “The Accumulation Buffer: Hardware Support for High-quality Rendering,” Proceedings of the 17th Annual Conference on Computer Graphics and Interactive Techniques, 1990, 10 pages. [cited by applicant]
He et al., “Deep Residual Learning for Image Recognition”, arXiv:1512.03385v1 [cs.CV], Dec. 10, 2015, pp. 1-12. [cited by applicant]
Iglesias-Guitian et al., “Pixel History Linear Models for Real-Time Temporal Filtering,” Computer Graphics Forum, 35(7): 2016, 10 pages. [cited by applicant]
Jaderberg et al., “Spatial Transformer Networks,” in Neural Information Processing Systems, 2015, 9 pages. [cited by applicant]
Jia et al., “Dynamic Filter Networks,” NeurIPS, 2016, 9 pages. [cited by applicant]
Jimenez et al., “Filmic SMAA: Sharp Morphological and Temporal Antialiasing4,” Advances in Real-Time Rendering in Games, Siggraph Courses, 2016, 122 pages. [cited by applicant]
Jimenez et al., “Filtering Approaches for Real-time Anti-aliasing,” ACM Siggraph, 2011, 14 pages. [cited by applicant]
Jimenez et al., “SMAA: Enhanced Subpixel Morphological Antialiasing,” Computer Graphics Forum, May 2012, 15 pages. [cited by applicant]
Jimenez et al., “SMAA: Enhanced Subpixel Morphological Antialiasing,” Eurographics, 2012, 15 pages. [cited by applicant]
Johnson et al., “Perceptual Losses for Real-Time Style Transfer and Super-Resolution,” ECCV, 2016, 17 pages. [cited by applicant]
Jégou et al., “The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation,” Dec. 5, 2016, 9 pages. [cited by applicant]
Karis et al., “High-Quality Temporal Supersampling,” Advances in Real-Time Rendering in Games, 2014, 55 pages. [cited by applicant]
Kerzner et al., “Streaming G-Buffer Compression for Multi-Sample Anti-Aliasing,” Proceedings of High Performance Graphics, 2014, 7 pages. [cited by applicant]
Kingma et al. “Adam: A Method for Stochastic Optimization,” arXiv:1412.6980, dated Dec. 22, 2014, 9 pages. [cited by applicant]
Lauritzen, “Deferred Rendering for Current and Future Rendering Pipelines,” Technical Report, Intel Corporation, 2010, 34 pages. [cited by applicant]
Lecun et al., “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE, 86(11): 1998, 47 pages. [cited by applicant]
Lottes, “FXAA,” Technical Report, Nvidia, Feb. 2009, 15 pages. [cited by applicant]
Malan, “Real-time Global Illumination and Reections in Dust 514,” Advances in Real-Time Rendering in Games, 2012, 51 pages. [cited by applicant]
Mammen,“Transparency and Antialiasing Algorithms Implemented with the Virtual Pixel Maps Technique,” IEEE Computer Graphics and Applications, 9(4): Jul. 1989, 13 pages. [cited by applicant]
Mao et al., “Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections,” Aug. 30, 2016, 17 pages. [cited by applicant]
Mara et al., “An Efficient Denoising Algorithm for Global Illumination,” Proceedings of High Performance Graphics, 2017, 7 pages. [cited by applicant]
Mildenhall et al., “Burst Denoising with Kernel Prediction Networks,” IEEE Conference on Computer Vision and Pattern Recognition, 2018, 9 pages. [cited by applicant]
Nalbach et al., “Deep Shading: Convolutional Neural Networks for Screen Space Shading,” Computer Graphics Forum, 36(4): 2016, 9 pages. [cited by applicant]
Nehab et al., “Accelerating Real-time Shading with Reverse Reprojection Caching,” Proceedings of the 22Nd ACM Siggraph/Eurographics Symposium on Graphics Hardware, 2007, 11 pages. [cited by applicant]
Niklaus et al., “Video Frame Interpolation via Adaptive Convolution,” CVPR, 2017, 10 pages. [cited by applicant]
Niklaus et al., “Video Frame Interpolation via Adaptive Separable Convolution,” ICCV, 2017, 10 pages. [cited by applicant]
Nilsson et al., “Semantic Video Segmentation by Gated Recurrent Flow Propagation,” Dec. 28, 2016, 10 pages. [cited by applicant]
Nvidia Research, “Nvidia OptiX AI-Accelerated Denoiser,” 2017, 11 pages. [cited by applicant]
Patney et al., “Towards Foveated Rendering for Gaze-Tracked Virtual Reality,” ACM Transactions on Graph 2016, 12 pages. [cited by applicant]
Ragan-Kelley et al., “The Lightspeed Automatic Interactive Lighting Preview System,” ACM Siggraph, 2007, 112 pages. [cited by applicant]
Reshetov et al., “Reducing Aliasing Artifacts Through Resampling,” Proceedings of the Fourth ACM Siggraph / Eurographics Conference on High-Performance Graphics, 2012, 10 pages. [cited by applicant]
Reshetov, “Morphological Antialiasing,” Proceedings of the Conference on High Performance Graphics, 2009, 8 pages. [cited by applicant]
U.S. Appl. No. 16/041,502, filed Jul. 20, 2018. [cited by applicant]
U.S. Appl. No. 18/225,674, filed Jul. 24, 2023. [cited by applicant]
U.S. Appl. No. 16/565,088, filed Sep. 9, 2019. [cited by applicant]
U.S. Appl. No. 17/543,075, filed Dec. 6, 2021. [cited by applicant]
U.S. Appl. No. 17/543,098, filed Dec. 6, 2021. [cited by applicant]
U.S. Appl. No. 18/219,594, filed Jul. 7, 2023. [cited by applicant]
Office Action for Australian Application No. 2023251540, mailed Jun. 7, 2024, 2 pages. [cited by applicant]
Office Action for Korean Application No. 10-2022-7015877, mailed Jun. 28, 2024, 5 pages. [cited by applicant]
Office Action for Australian Application No. 2023251547, mailed Jun. 18, 2024, 3 pages. [cited by applicant]
Office Action for Australian Application No. 2023251542, mailed Jun. 18, 2024, 3 pages. [cited by applicant]
Office Action for United Kingdom Application No. GB2115845.6, mailed Jul. 9, 2024, 3 pages. [cited by applicant]
U.S. Appl. No. 17/095,556, filed Nov. 11, 2020. [cited by applicant]
U.S. Appl. No. 17/089,445, filed Nov. 4, 2020. [cited by applicant]
U.S. Appl. No. 17/406,902, filed Aug. 19, 2021. [cited by applicant]
U.S. Appl. No. 17/710,643, filed Mar. 31, 2022. [cited by applicant]
U.S. Appl. No. 17/080,503, filed Oct. 26, 2020. [cited by applicant]
Office Action for Japanese Application No. 2022-526732,mailed Sep. 5, 2024, 4 pages. [cited by applicant]
Office Action for Japanese Application No. 2022-525355, mailed Sep. 10, 2024, 7 pages. [cited by applicant]
Office Action for Japanese Application No. 2023-188130, mailed Oct. 10, 2024, 9 pages. [cited by applicant]
Office Action for Japanese Application No. 2023-188131, mailed Oct. 8, 2024, 6 pages. [cited by applicant]
Office Action for United Kingdom Application No. GB2205245.0, mailed Sep. 3, 2024, 2 pages. [cited by applicant]
Office Action for United Kingdom Application No. GB2316422.1, mailed Mar. 7, 2024, 4 pages. [cited by applicant]
Chu et al., “Temporally Coherent GANs for Video Super-Resolution (TecoGAN),” Nov. 23, 2018, 17 pages. [cited by applicant]
Jordan, “Neural Networks: Training with Backpropagation,” Data Science, Jul. 18, 2017, 32 pages. [cited by applicant]
Kappeler et al., “Video Super-Resolution With Convolutional Neural Networks,” IEEE Transactions on Computational Imaging, 2(2): Jun. 1, 2016, 14 pages. [cited by applicant]
Kong et al., “A Conditional Random Field Model for Video Super-Resolution,” International Conference on Pattern Recognition, Jan. 1, 2006, 4 pages. [cited by applicant]
Li et al., “Fast Spatio-Temporal Residual Network for Video Super-Resolution,” CVPR, 2019, 10 pages. [cited by applicant]
Office Action for United Kingdom Application No. GB2115845.6, mailed Apr. 11, 2023, 4 pages. [cited by applicant]
Torres et al., “An Efficient Approach to Automatic Generation of Time-lapse Video Sequences,” Proceedings of the 21st Irish Machine Vision and Image Processing Conference, 2019, 8 pages. [cited by applicant]
United Kingdom Combined Search and Examination Report for Patent Application No. 2115845.6 dated Jul. 7, 2022, 10 pages. [cited by applicant]
Wang et al., “Deep Learning for Image Super-Resolution: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10): Mar. 23, 2020, 23 pages. [cited by applicant]
Wang et al., “Video Super-Resolution via Residual Learning,” May 14, 2018, 11 pages. [cited by applicant]
International Search Report and Written Opinion for Application No. PCT/US2021/056477, mailed Feb. 9, 2022, filed Oct. 25, 2021, 14 pages. [cited by applicant]
Marín et al., “Learning Appearance in Virtual Scenarios for Pedestrian Detection,” IEEE Conference on Computer Vision and Pattern Recognition, Jun. 13, 2010, 8 pages. [cited by applicant]
Rematas et al., “Image-Based Synthesis and Re-synthesis of Viewpoints Guided by 3D Models,” IEEE Conference on Computer Vision and Pattern Recognition, Jun. 23, 2014, 8 pages. [cited by applicant]
Rozantsev et al., “On Rendering Synthetic Images for Training an Object Detector,” Computer Vision and Image Understanding Academic Press, vol. 137, Jan. 20, 2015, 30 pages. [cited by applicant]
Hedman et al., “Deep Blending for Free-Viewpoint Image-Based Rendering,” Siggraph Asia Technical Papers, 2018, 15 pages. [cited by applicant]
United Kingdom Combined Search and Examination Report for Patent Application No. 2116203.7 dated Jul. 12, 2022, 12 pages. [cited by applicant]
Office Action for Japanese Application No. 2022-515469, mailed Apr. 18, 2024, 6 pages. [cited by applicant]
Manabe et al., “FPGA Implementation of a Video Super-Resolution System,” IEICE Technical Report, May 12, 2016, 7 pages. [cited by applicant]
Office Action for Korean Application No. 10-2024-7013804, mailed May 20, 2024, 6 pages. [cited by applicant]
Office Action for Korean Application No. 10-2024-7013803, mailed May 20, 2024, 5 pages. [cited by applicant]
Caballero et al., “Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation,” 2017, 10 pages. [cited by applicant]
Chen et al., “Coherent Online Video Style Transfer,” 2017, 10 pages. [cited by applicant]
Combined Search and Examination Report for United Kingdom Application No. GB2311266.7, mailed Feb. 21, 2024, 4 pages. [cited by applicant]
Combined Search and Examination Report for United Kingdom Application No. GB2316421.3, mailed Feb. 26, 2024, 3 pages. [cited by applicant]
Lore et al., “LLNet: A Deep Autoencoder Approach to Natural Low-light Image Enhancement,” 2016, 11 pages. [cited by applicant]
Office Action for German Application No. 10-218-117813.1, mailed Feb. 27, 2024, 14 pages. [cited by applicant]
Office Action for Japanese Application No. 2022-515469, mailed Nov. 8, 2024, 2 pages. [cited by applicant]
Office Action for Korean Patent Application No. 10-2024-7013803, mailed Nov. 22, 2024, 3 pages. [cited by applicant]
Office Action for Korean Patent Application No. 10-2024-7013804, mailed Nov. 22, 2024, 3 pages. [cited by applicant]
Office Action for Korean Patent Application No. 10-2024-7013805, mailed Nov. 22, 2024, 3 pages. [cited by applicant]
Office Action for Chinese Application No. 202111301494.3, mailed Feb. 12, 2025, 20 pages. [cited by applicant]
Office Action for Chinese Application No. 202111314003.9, mailed Feb. 12, 2025, 17 pages. [cited by applicant]
Office Action for Japanese Application No. 2023-188129, mailed Oct. 8, 2024, 12 pages. [cited by applicant]
Craven, “A Faster Learning Neural Network Classifier Using Selective Backpropagation,” IEEE International Conference on Electronics, Circuits and Systems, 1997, 5 pages. [cited by applicant]
Examination Report for United Kingdom Application No. GB2115845.6, mailed Apr. 15, 2025, 5 pages. [cited by applicant]
Examination Report for United Kingdom Application No. GB2115845.6, mailed Jun. 5, 2025, 5 pages. [cited by applicant]
Examination Report for United Kingdom Application No. GB2205245.0, mailed Apr. 23, 2025, 6 pages. [cited by applicant]
Jiang et al., “Accelerating Deep Learning by Focusing on the Biggest Losers,” 2019, 14 pages. [cited by applicant]
Office Action for Chinese Application No. 202080062960.4, mailed Apr. 27, 2025, 17 pages. [cited by applicant]
Oh et al., “Learning-based Video Motion Magnification”, Aug. 1, 2018, 26 pages. [cited by applicant]
Examination Report for United Kingdom Application No. GB2115845.6, mailed Jul. 2, 2025, 4 pages. [cited by applicant]
Notice of Decision to Grant for Chinese Application No. 202180005531.8, mailed Jul. 11, 2025, 8 pages. [cited by applicant]
Office Action for Chinese Application No. 202111301494.3, mailed Jul. 9, 2025, 21 pages. [cited by applicant]
Examination Report for United Kingdom Application No. GB2115845.6, mailed Aug. 7, 2025, 3 pages. [cited by applicant]
Office Action for Chinese Application No. 202310920699.2, mailed Aug. 6, 2025, 14 pages. [cited by applicant]
Peng et al., “Deep Deconvolution Neural Network for Image Super-Resolution” 2017, 9 pages. [cited by applicant]
Office Action for Chinese Application No. 202111314003.9, mailed Jul. 31, 2025, 17 pages. [cited by applicant]
Office Action for Chinese Application No. 202180005531.8, mailed Mar. 5, 2025, 18 pages. [cited by applicant]
Office Action for Chinese Application No. 202111301494.3, mailed Oct. 15, 2025, 19 pages. [cited by applicant]
Decision of Rejection for Chinese Application No. 202111314003.9, mailed Oct. 31, 2025, 13 pages. [cited by applicant]
Office Action for Chinese Application No. 20211160114.3, mailed Dec. 2, 2025, 16 pages. [cited by applicant]
Office Action for Chinese Application No. 202080062960.4, mailed Dec. 3, 2025, 20 pages. [cited by applicant]
Office Action for Chinese Application No. 202211160111.X, mailed Dec. 23, 2025, 16 pages. [cited by applicant]