IP Library › Granted Patent US 12,561,757
Granted Patent B2
US 12,561,757 · App. 18/379,519 · Granted Feb 24, 2026

Image super-resolution neural networks

Inventors: Cristina Nader Vasconcelos (Montreal, CA); Ahmet Cengiz Oztireli (Zurich, CH); Andrea Tagliasacchi (Toronto, CA); Kevin Jordan Swersky (Toronto, CA); Mark Jeffrey Matthews (Los Angeles, CA); Milad Olia Hashemi (San Francisco, CA)
Assignee: Google LLC
G06T3/4053G06T5/20G06V10/771G06T2207/10024G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,561,757
App. No.
18/379,519
Granted
Feb 24, 2026
Kind
B2
Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing an input image using a super-resolution neural network to generate an up-sampled image that is a higher resolution version of the input image. In one aspect, a method comprises: processing the input image using an encoder subnetwork of the super-resolution neural network to generate a feature map; generating an updated feature map, comprising, for each spatial position in the updated feature map: applying a convolutional filter to the feature map to generate a plurality of features corresponding to the spatial position in the updated feature map, wherein the convolutional filter is parametrized by a set of convolutional filter parameters that are generated by processing data representing the spatial position using a hyper neural network; and processing the updated feature map using a projection subnetwork of the super-resolution neural network to generate the up-sampled image.

Claims (53)

1 . A method performed by one or more computers, the method comprising:

receiving an input image and an up-sampling factor;

processing the input image using a super-resolution neural network to generate an up-sampled image that is a higher resolution version of the input image, comprising:

processing the input image using an encoder subnetwork of the super-resolution neural network to generate a feature map;

generating an updated feature map, comprising, for each spatial position in the updated feature map:

applying a convolutional filter to the feature map at the spatial position to generate a plurality of features corresponding to the spatial position in the updated feature map,

wherein the convolutional filter is parametrized by a set of convolutional filter parameters that are generated by processing data comprising (i) data representing the spatial position, and (ii) the up-sampling factor, using a hyper neural network; and

processing the updated feature map using a projection subnetwork of the super-resolution neural network to generate the up-sampled image.

2 . The method of claim 1 , wherein the hyper neural network is configured to receive an input comprising an input spatial position, wherein the input spatial position is selected from a continuous space of possible spatial positions.

3 . The method of claim 1 , wherein the up-sampling factor is selected from a continuous range of possible up-sampling factors.

4 . The method of claim 1 , wherein the up-sampling factor defines a ratio of: (i) a resolution of the up-sampled image, and (ii) a resolution of the input image.

5 . The method of claim 1 , wherein the hyper neural network is configured to receive an input that comprises an index of one or more convolutional filter parameters; and

wherein the hyper neural network is configured to generate an output that comprises one or more convolutional filter parameters corresponding to the index included in the input to the hyper neural network.

6 . The method of claim 5 , wherein processing data comprising (i) data representing the spatial position, and (ii) the up-sampling factor, using the hyper neural network to generate the convolutional filter comprises, for each convolutional filter parameter in the convolutional filter:

processing an input comprising: (i) data representing the spatial position, and (ii) an index corresponding to the convolutional filter parameter, using the hyper neural network to generate the convolutional filter parameter corresponding to the index.

7 . The method claim 1 , wherein the hyper neural network is configured to apply positional encoding to inputs to the hyper neural network.

8 . The method of claim 7 , wherein the positional encoding is a cosine positional encoding.

9 . The method of claim 1 , wherein for each spatial position in the updated feature map, the convolutional filter corresponding to the spatial position is a two-dimensional convolutional filter.

10 . The method of claim 1 , wherein processing the input image using the encoder subnetwork of the super-resolution neural network to generate the feature map comprises:

processing the input image to generate a feature map having a same resolution as the input image; and

transforming the feature map to have a same resolution as the up-sampled image.

11 . The method of claim 10 , wherein transforming the feature map to have the same resolution as the up-sampled image comprises:

up-sampling the feature map using nearest-neighbor interpolation.

12 . The method of claim 10 , wherein processing the input image to generate a feature map having a same resolution as the input image comprises:

unfolding the feature map to augment each spatial position in the feature map with features from neighboring spatial positions in the feature map.

13 . The method of claim 1 , wherein processing the updated feature map using the projection subnetwork of the super-resolution neural network to generate the up-sampled image comprises, for each spatial position in the updated feature map:

processing the plurality of features corresponding to the spatial position in the updated feature map using the projection subnetwork to generate one or more intensity values of a pixel at the spatial position in the up-sampled image.

14 . The method of claim 1 , wherein input image is a two-dimensional color image.

15 . The method of claim 1 , wherein the input image comprises a medical image.

16 . The method of claim 1 , further comprising:

determining gradients of a super-resolution objective function with respect to: (i) a set of parameters of the super-resolution neural network, and (ii) a set of parameters of the hyper neural network, wherein the super-resolution objective function measures an error in the up-sampled image; and

updating current values of: (i) the set of parameters of the super-resolution neural network, and (ii) the set of parameters of the hyper neural network, using the gradients.

17 . The method of claim 16 , wherein updating current values of: (i) the set of parameters of the super-resolution neural network, and (ii) the set of parameters of the hyper neural network, using the gradients comprises:

backpropagating the gradients through the super-resolution neural network and the hyper neural network.

18 . A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

receiving an input image and an up-sampling factor;

processing the input image using a super-resolution neural network to generate an up-sampled image that is a higher resolution version of the input image, comprising:

processing the input image using an encoder subnetwork of the super-resolution neural network to generate a feature map;

generating an updated feature map, comprising, for each spatial position in the updated feature map:

applying a convolutional filter to the feature map to generate a plurality of features corresponding to the spatial position in the updated feature map,

wherein the convolutional filter is parametrized by a set of convolutional filter parameters that are generated by processing data comprising (i) data representing the spatial position, and (ii) the up-sampling factor, using a hyper neural network; and

processing the updated feature map using a projection subnetwork of the super-resolution neural network to generate the up-sampled image.

19 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving an input image and an up-sampling factor;

processing the input image using a super-resolution neural network to generate an up-sampled image that is a higher resolution version of the input image, comprising:

processing the input image using an encoder subnetwork of the super-resolution neural network to generate a feature map;

generating an updated feature map, comprising, for each spatial position in the updated feature map:

applying a convolutional filter to the feature map to generate a plurality of features corresponding to the spatial position in the updated feature map,

wherein the convolutional filter is parametrized by a set of convolutional filter parameters that are generated by processing data comprising (i) data representing the spatial position, and (ii) the up-sampling factor, using a hyper neural network; and

processing the updated feature map using a projection subnetwork of the super-resolution neural network to generate the up-sampled image.

20 . The one or more non-transitory computer storage media of claim 19 , wherein the hyper neural network is configured to receive an input comprising an input spatial position, wherein the input spatial position is selected from a continuous space of possible spatial positions.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 19, 2023
From: NADER VASCONCELOS, CRISTINA; OZTIRELI, AHMET CENGIZ; TAGLIASACCHI, ANDREA; SWERSKY, KEVIN JORDAN; MATTHEWS, MARK JEFFREY; HASHEMI, MILAD OLIA
To: GOOGLE LLC
Reel/Frame 065276/0644 →
Continuity (2)
Provisional Application 63415551 · Oct 12, 2022
Related Publication 20240135492A1 · Apr 25, 2024
References Cited (51)
US 11288546B2 · Lee · 2022 [cited by examiner]
US 11875553B2 · Won · 2024 [cited by examiner]
US 20170185871A1 · Zhang · 2017 [cited by examiner]
US 20190080456A1 · Song · 2019 [cited by examiner]
US 20190164290A1 · Wang · 2019 [cited by examiner]
US 20190295228A1 · Liu · 2019 [cited by examiner]
US 20200211229A1 · Hwang · 2020 [cited by examiner]
CN 111062867A · 2020 [cited by examiner]
CN 112364981A · 2021 [cited by examiner]
CN 109447247B · 2021 [cited by examiner]
KR 102337412B1 · 2021 [cited by examiner]
Alwani et al., “Fused-layer CNN Accelerators,” Paper, Presented at 49th Annual IEEE/ACM International Symposium on Microarchitecture, Taipei, Taiwan, Oct. 15-19, 2016, 12 pages. [cited by applicant]
Anwar et al., “A deep journey into super-resolution: A survey,” ACM Computing Surveys, May 2020, 53(3):1-34. [cited by applicant]
Bashir et al., “A comprehensive review of deep learning based single image super-resolution,” PeerJ Computer Science, Jul. 2021, 7:e621, pp. 1-56. [cited by applicant]
Bevilacqua et al., “Low-complexity single-image super-resolution based on nonnegative neighbor embedding,” Paper, Presented at Proceedings of the British Machine Vision Conference, Surrey, United Kingdom, Sep. 3-7, 2012… [cited by applicant]
Boulch, “ConvPoint: Continuous convolutions for point cloud processing,” Computers & Graphics, May 2020, 88:24-34. [cited by applicant]
Chen et al., “Learning continuous image representation with local implicit image function,” Paper, Presented at Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, Jun. 20-2… [cited by applicant]
Dai et al., “Second-order attention network for single image super-resolution,” Paper, Presented at 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, Jun. 15-20, 2019, pp. 11065-11074. [cited by applicant]
Dong et al., “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Feb. 2016, 38(2):1-14. [cited by applicant]
Dong et al., “Learning a deep convolutional network for image super-resolution,” Paper, Presented at Computer Vision—European Conference on Computer Vision 2014, Zurich, Switzerland, Sep. 6-12, 2014; Lecture Notes in Co… [cited by applicant]
Du et al., “Anchor-Based Plain Net for Mobile Image Super-Resolution,” Paper, Presented at 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Nashville, TN, Jun. 19-25, 2021, pp. 1-9. [cited by applicant]
Github.io [online], “CUF: Continuous Upsampling Filters,” Jun. 2023, retrieved on Nov. 19, 2024, retrieved from URL<https://cuf-paper.github.io/>, 5 pages. [cited by applicant]
Goodfellow et al., “Generative adversarial networks,” Communications of the ACM, Oct. 2020, 63(11):139-144. [cited by applicant]
Ha et al., “HyperNetworks,” CoRR, Submitted on Dec. 1, 2016, arXiv:1609.09106v4, pp. 1-29. [cited by applicant]
Hermosilla et al., “Monte Carlo convolution for learning on non-uniformly sampled point clouds,” ACM Transactions on Graphics, Dec. 2018, 37(6):1-12. [cited by applicant]
Ho et al., “Denoising diffusion probabilistic models,” Paper, Presented at 34th Conference on Neural Information Processing Systems, Vancouver, Canada, Dec. 6-12, 2020; Advances in Neural Information Processing Systems … [cited by applicant]
Hu et al., “Meta-SR: A magnification-arbitrary network for super-resolution,” Paper, Presented at Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, Jun. 15-20, 2019, pp. 1575… [cited by applicant]
Huang et al., “Single image super-resolution from transformed self-exemplars,” Paper, Presented at Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, Jun. 7-12, 2015, pp. 519… [cited by applicant]
Ignatov et al., “Real-Time Quantized Image Super-Resolution on Mobile NPUs, Mobile AI 2021 Challenge: Report,” Paper, Presented at Computer Vision and Pattern Recognition, 2021 Mobile AI Workshop, Virtual, Jun. 20, 2021… [cited by applicant]
Lee et al., “Local texture estimator for implicit representation function,” Paper, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, Jun. 18-24, 2022, pp. 1929-1938. [cited by applicant]
Liang et al., “SwinIR: Image Restoration Using Swin Transformer,” Paper, Presented at IEEE International Conference on Computer Vision Workshops, Montreal, Canada, Oct. 11-17, 2021, pp. 1833-1844. [cited by applicant]
Lim et al., “Enhanced Deep Residual Networks for Single Image Super-Resolution,” Paper, Presented at the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, Jul. 21-26, 2017, pp. 136-144. [cited by applicant]
Liu et al., “Swin transformer: Hierarchical vision transformer using shifted windows,” Paper, Presented at Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, Canada, Oct. 10-17, 2021, pp.… [cited by applicant]
Martin et al., “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” Paper, Presented in Proceedings of the Eighth IEEE Internation… [cited by applicant]
Mildenhall et al., “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis,” Paper, Presented at European Conference on Computer Vision, Glasgow, Scotland, Aug. 23-28, 2020; Lecture Notes in Computer Sci… [cited by applicant]
Rahaman et al., “On the Spectral Bias of Neural Networks,” Paper, Presented at Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, Jun. 9-15, 2019; PMLR 97:5301-5310, 10 pages. [cited by applicant]
Rahimi et al., “Random features for largescale kernel machines,” Paper, Presented at 21st Conference on Neural Information Processing Systems, Vancouver, Canada, Dec. 3-6, 2007; Advances in Neural Information Processing… [cited by applicant]
Romero et al., “CKconv: Continuous Kernel Convolution for Sequential Data,” Paper, Presented at the Tenth International Conference on Learning Representations, Virtual Conference, Apr. 25-29, 2022, pp. 1-25. [cited by applicant]
Romero et al., “Flexconv: Continuous Kernel Convolutions with Differentiable Kernel Sizes,” Paper, Presented at Tenth International Conference on Learning Representations, Virtual Conference, Apr. 25-29, 2022, pp. 1-24. [cited by applicant]
Saharia, “Image Super-Resolution via Iterative Refinement,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Sep. 12, 2022, 45(4):4713-4726. [cited by applicant]
Shi et al., “Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network,” Paper, Presented at IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, J… [cited by applicant]
Tancik et al., “Fourier features let networks learn high frequency functions in low dimensional domains,” Paper, Presented at 34th Conference on Neural Information Processing Systems, Vancouver, Canada, Dec. 6-12, 2020;… [cited by applicant]
Timofte et al., “NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results,” Paper, Presented at Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, Ju… [cited by applicant]
Vaswani et al., “Attention is all you need,” Paper, Presented at 31st Conference on Neural Information Processing Systems, Long Beach, CA, Dec. 4-9, 2017; Advances in Neural Information Processing Systems 30, Dec. 2017,… [cited by applicant]
Wang et al., “Deep Learning or image super-resolution: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Oct. 2021, 43(10):3365-3387. [cited by applicant]
Wang et al., “ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks,” Paper, Presented at Computer Vision—European Conference on Computer Vision 2018 Workshops, Munich, Germany, Sep. 8-14, 2018; Lecture Note… [cited by applicant]
Wu et al., “PointConv: Deep convolutional networks on 3d point clouds, ” Paper, Presented at IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, Jun. 16-20, 2019, pp. 9621-9630. [cited by applicant]
Xie et al., “Neural fields in visual computing and beyond,” CoRR, Submitted on Nov. 29, 2021, arXiv:2111.11426v3, 31 pages. [cited by applicant]
Zeyde et al., “On single image scale-up using sparse-representations,” Paper, Presented at the 7th International Conference, Avignon, France, Jun. 24-30, 2010: Lecture Notes in Computer Science, 2011, 6920:711-730. [cited by applicant]
Zhang et al., “Image super-resolution using very deep residual channel attention networks,” Paper, Presented at the Computer Vision—European Conference on Computer Vision 2018, Munich, Germany, Sep. 8-14, 2018; Lecture … [cited by applicant]
Zhang et al., “Residual dense network for image restoration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Jan. 2020, 43(7):2480-2495. [cited by applicant]