IP Library › Granted Patent US 12,450,690
Granted Patent B2
US 12,450,690 · App. 17/987,491 · Granted Oct 21, 2025

Method, device, and computer program product for image processing

Inventors: Zhisong Liu (Shenzhen, CN); Zijia Wang (WeiFang, CN); Zhen Jia (Shanghai, CN)
Assignee: Dell Products L.P.
G06T3/4046G06N3/08G06T3/4053
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,450,690
App. No.
17/987,491
Granted
Oct 21, 2025
Kind
B2
Abstract

A method includes generating a feature map of a second resolution for an input image of a first resolution, the first resolution being less than the second resolution. The method further includes generating a feature map of the second resolution and a feature map of the first resolution through a first residual-based dense back projection (RDBP) network. In addition, the method further includes generating a feature map of the second resolution and a feature map of the first resolution through a second RDBP network, the first RDBP network and the second RDBP network respectively including a downsampling back projection layer, an upsampling back projection layer, and an attention layer. The method further includes generating an output image of the second resolution based on the feature maps of the second resolution and the feature maps of the first resolution generated through the first RDBP network and the second RDBP network.

Claims (66)

1. A method for image processing, comprising:

for an input image of a first resolution, generating a feature map of a second resolution, the first resolution being less than the second resolution;

processing the feature map of the second resolution through a processing pipeline comprising at least a first residual-based dense back projection (RDBP) network, a second RDBP network, and a cross-scale attention layer, the first RDBP network and the second RDBP network each comprising a downsampling back projection layer, an upsampling back projection layer, and a spatial attention layer, the first and second RDBP networks being configured to generate respective instances of the feature map of the second resolution and respective corresponding instances of a feature map of the first resolution, for processing by the cross-scale attention layer of the processing pipeline, the cross-scale attention layer comprising a multi-head attention block including a bottom-up attention block and a top-down attention block, the bottom-up attention block and the top-down attention block generating respective feature maps of respective different resolutions with attention adjusted across scales by processing outputs of different ones of the downsampling back projection layers and the upsampling back projection layers of the first and second RDBP networks; and

generating, utilizing the cross-scale attention layer of the processing pipeline, an output image of the second resolution based on the feature maps of the second resolution and the feature maps of the first resolution generated through the first RDBP network and the second RDBP network.

2. The method according to claim 1 , wherein generating a feature map of a second resolution for an input image of a first resolution comprises:

performing convolving and weighting on the input image to extract the feature map of the first resolution; and

upsampling the feature map of the first resolution to obtain the feature map of the second resolution.

3. The method according to claim 1 , wherein generating a feature map of the second resolution and a feature map of the first resolution through a first RDBP network comprises:

downsampling the feature map of the second resolution to obtain an initial downsampled feature map;

upsampling and downsampling the initial downsampled feature map to generate a first upsampled feature map and a first downsampled feature map; and

performing weighting on the initial downsampled feature map and the first downsampled feature map to generate a first adjusted feature map.

4. The method according to claim 3 , wherein generating a feature map of the second resolution and a feature map of the first resolution through a first RDBP network further comprises:

upsampling and downsampling the first adjusted feature map to generate a second upsampled feature map and a second downsampled feature map; and

performing weighting on the initial downsampled feature map, the first downsampled feature map, and the second downsampled feature map to generate a second adjusted feature map.

5. The method according to claim 4 , wherein generating a feature map of the second resolution and a feature map of the first resolution through a first RDBP network further comprises:

upsampling the second adjusted feature map to generate a third upsampled feature map;

convolving the first upsampled feature map, the second upsampled feature map, and the third upsampled feature map to generate an attention-adjusted feature map; and

determining the attention-adjusted feature map to be the feature map of the second resolution, and determining the second adjusted feature map to be the feature map of the first resolution.

6. The method according to claim 2 , wherein the upsampling comprises:

deconvolving and convolving the feature map of the first resolution to obtain a convolved feature map; and

generating the feature map of the second resolution based on the feature map of the first resolution and the convolved feature map.

7. The method according to claim 3 , wherein the downsampling comprises:

convolving and deconvolving the feature map of the second resolution to obtain a deconvolved feature map; and

generating the feature map of the first resolution based on the feature map of the second resolution and the deconvolved feature map.

8. The method according to claim 5 , further comprising:

iteratively performing a process of generating the second adjusted feature map prior to generating the third upsampled feature map.

9. The method according to claim 1 , further comprising:

iteratively performing a process of generating the feature map of the second resolution and the feature map of the first resolution through an RDBP network multiple times.

10. The method according to claim 1 , wherein generating an output image of the second resolution comprises:

connecting a plurality of feature maps of the second resolution generated through the first RDBP network and the second RDBP network together to generate a feature map set of the second resolution; and

connecting a plurality of feature maps of the first resolution generated through the first RDBP network and the second RDBP network together to generate a feature map set of the first resolution.

11. The method according to claim 10 , wherein generating an output image of the second resolution further comprises:

performing first weighting on the feature map set of the second resolution and the feature map set of the first resolution to generate a feature map of the second resolution with attention adjusted across scales; and

performing second weighting on the feature map set of the second resolution and the feature map set of the first resolution to generate a feature map of the first resolution with attention adjusted across scales.

12. The method according to claim 11 , wherein generating an output image of the second resolution further comprises:

generating the output image of the second resolution based on the feature map of the second resolution with attention adjusted across scales, the feature map of the first resolution with attention adjusted across scales, and the input image of the first resolution.

13. An electronic device, comprising:

at least one processor; and

memory coupled to the at least one processor and having instructions stored thereon, wherein the instructions, when executed by the at least one processor, cause the electronic device to perform actions comprising:

for an input image of a first resolution, generating a feature map of a second resolution, the first resolution being less than the second resolution;

processing the feature map of the second resolution through a processing pipeline comprising at least a first residual-based dense back projection (RDBP) network, a second RDBP network, and a cross-scale attention layer, the first RDBP network and the second RDBP network each comprising a downsampling back projection layer, an upsampling back projection layer, and a spatial attention layer, the first and second RDBP networks being configured to generate respective instances of the feature map of the second resolution and respective corresponding instances of a feature map of the first resolution, for processing by the cross-scale attention layer of the processing pipeline, the cross-scale attention layer comprising a multi-head attention block including a bottom-up attention block and a top-down attention block, the bottom-up attention block and the top-down attention block generating respective feature maps of respective different resolutions with attention adjusted across scales by processing outputs of different ones of the downsampling back projection layers and the upsampling back projection layers of the first and second RDBP networks; and

generating, utilizing the cross-scale attention layer of the processing pipeline, an output image of the second resolution based on the feature maps of the second resolution and the feature maps of the first resolution generated through the first RDBP network and the second RDBP network.

14. The electronic device according to claim 13 , wherein generating a feature map of a second resolution for an input image of a first resolution comprises:

performing convolving and weighting on the input image to extract the feature map of the first resolution; and

upsampling the feature map of the first resolution to obtain the feature map of the second resolution.

15. The electronic device according to claim 13 , wherein generating a feature map of the second resolution and a feature map of the first resolution through a first RDBP network comprises:

downsampling the feature map of the second resolution to obtain an initial downsampled feature map;

upsampling and downsampling the initial downsampled feature map to generate a first upsampled feature map and a first downsampled feature map; and

performing weighting on the initial downsampled feature map and the first downsampled feature map to generate a first adjusted feature map.

16. The electronic device according to claim 15 , wherein generating a feature map of the second resolution and a feature map of the first resolution through a first RDBP network further comprises:

upsampling and downsampling the first adjusted feature map to generate a second upsampled feature map and a second downsampled feature map; and

performing weighting on the initial downsampled feature map, the first downsampled feature map, and the second downsampled feature map to generate a second adjusted feature map.

17. The electronic device according to claim 16 , wherein generating a feature map of the second resolution and a feature map of the first resolution through a first RDBP network further comprises:

upsampling the second adjusted feature map to generate a third upsampled feature map;

convolving the first upsampled feature map, the second upsampled feature map, and the third upsampled feature map to generate an attention-adjusted feature map; and

determining the attention-adjusted feature map to be the feature map of the second resolution, and determining the second adjusted feature map to be the feature map of the first resolution.

18. The electronic device according to claim 14 , wherein the upsampling comprises:

deconvolving and convolving the feature map of the first resolution to obtain a convolved feature map; and

generating the feature map of the second resolution based on the feature map of the first resolution and the convolved feature map.

19. The electronic device according to claim 15 , wherein the downsampling comprises:

convolving and deconvolving the feature map of the second resolution to obtain a deconvolved feature map; and

generating the feature map of the first resolution based on the feature map of the second resolution and the deconvolved feature map.

20. A computer program product that is tangibly stored on a non-transitory computer-readable medium and comprises machine-executable instructions, wherein the machine-executable instructions, when executed by a machine, cause the machine to perform actions comprising:

for an input image of a first resolution, generating a feature map of a second resolution, the first resolution being less than the second resolution;

processing the feature map of the second resolution through a processing pipeline comprising at least a first residual-based dense back projection (RDBP) network, a second RDBP network, and a cross-scale attention layer, the first RDBP network and the second RDBP network each comprising a downsampling back projection layer, an upsampling back projection layer, and a spatial attention layer, the first and second RDBP networks being configured to generate respective instances of the feature map of the second resolution and respective corresponding instances of a feature map of the first resolution, for processing by the cross-scale attention layer of the processing pipeline, the cross-scale attention layer comprising a multi-head attention block including a bottom-up attention block and a top-down attention block, the bottom-up attention block and the top-down attention block generating respective feature maps of respective different resolutions with attention adjusted across scales by processing outputs of different ones of the downsampling back projection layers and the upsampling back projection layers of the first and second RDBP networks; and

generating, utilizing the cross-scale attention layer of the processing pipeline, an output image of the second resolution based on the feature maps of the second resolution and the feature maps of the first resolution generated through the first RDBP network and the second RDBP network.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 15, 2022
From: LIU, ZHISONG; WANG, ZIJIA; JIA, ZHEN
To: DELL PRODUCTS L.P.
Reel/Frame 061780/0801 →
Priority Claims (1)
CN 202211298887.8 · Oct 21, 2022 · national
Continuity (2)
Related Publication 20240135489A1 · Apr 25, 2024
Related Publication 20240233073A9 · Jul 11, 2024
References Cited (56)
US 11010871B1 · Lee · 2021 [cited by examiner]
US 11983239B2 · Lian · 2024 [cited by examiner]
US 12190520B2 · Nie · 2025 [cited by examiner]
US 12212764B2 · Gao · 2025 [cited by examiner]
US 12313732B2 · Chen · 2025 [cited by examiner]
US 20210133925A1 · Lee · 2021 [cited by examiner]
US 20220286696A1 · Gao · 2022 [cited by examiner]
US 20220391635A1 · Lian · 2022 [cited by examiner]
US 20220391636A1 · Lian · 2022 [cited by examiner]
US 20230184927A1 · Chen · 2023 [cited by examiner]
US 20230196745A1 · Hur · 2023 [cited by examiner]
US 20230400301A1 · Song · 2023 [cited by examiner]
US 20240013399A1 · Nie · 2024 [cited by examiner]
CN 109741260A · 2019 [cited by examiner]
CN 110276721A · 2019 [cited by examiner]
CN 111192200A · 2020 [cited by examiner]
CN 111353940A · 2020 [cited by examiner]
CN 112699848A · 2021 [cited by examiner]
CN 115063435A · 2022 [cited by examiner]
WO WO2021104060A1 · 2021 [cited by examiner]
CN-109741260-A (machine translation) (Year: 2019). [cited by examiner]
WO-2021104060-A1 (machine translation) (Year: 2021). [cited by examiner]
Xu et al., “Sparse cross-scale attention network for efficient lidar panoptic segmentation.” In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, No. 3, pp. 2920-2928. 2022. (Year: 2022). [cited by examiner]
Deng et al., “Cross-scale attention guided multi-instance learning for Crohn's disease diagnosis with pathological images.” In International workshop on multiscale multimodal medical imaging, pp. 24-33. Cham: Springer N… [cited by examiner]
Mei et al., “Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining.” In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5690-5699. 2020. … [cited by examiner]
CN-110276721-A (machine translation) (Year: 2019). [cited by examiner]
CN-111192200-A (machine translation) (Year: 2020). [cited by examiner]
CN-111353940-A (machine translation) (Year: 2020). [cited by examiner]
CN-112699848-A (machine translation) (Year: 2021). [cited by examiner]
CN-115063435-A (machine translation) (Year: 2022). [cited by examiner]
Xu et al., “Spatial—spectral FFPNet: Attention-Based Pyramid Network for Segmentation and Classification of Remote Sensing Images.” arXiv preprint arXiv:2008.08775 (2020). (Year: 2020). [cited by examiner]
Deng et al., “Cross-scale Attention Guided Multi-instance Learning for Crohn's Disease Diagnosis with Pathological Images.” arXiv preprint arXiv:2208.07322 (2022). (Year: 2022). [cited by examiner]
Nie et al., “Pyramid Architecture for Multi-Scale Processing in Point Cloud Segmentation,” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 17263-17273 (Year: 2… [cited by examiner]
Han et al., “3CROSSNet: Cross-Level Cross-Scale Cross-Attention Network for Point Cloud Representation,” in IEEE Robotics and Automation Letters, vol. 7, No. 2, pp. 3718-3725, Apr. 2022 (Year: 2022). [cited by examiner]
Ni et al., “Dual Path Cross-Scale Attention Network for Image Inpainting,” 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France, 2022, pp. 4223-4227 (Year: 2022). [cited by examiner]
Lin et al., “BATFormer: Towards Boundary-Aware Lightweight Transformer for Efficient Medical Image Segmentation.” arXiv preprint arXiv:2206.14409 (2022). (Year: 2022). [cited by examiner]
He et al., “ForkNet: Strong Semantic Feature Representation and Subregion Supervision for Accurate Remote Sensing Change Detection,” in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, v… [cited by examiner]
Qiu et al., “IVT: An End-to-End Instance-guided Video Transformer for 3D Pose Estimation.” arXiv preprint arXiv:2208.03431 (2022). (Year: 2022). [cited by examiner]
Lu et al., “Cross-scale attention model for acoustic event classification.” arXiv preprint arXiv:1912.12011v2 (2020). (Year: 2020). [cited by examiner]
Bashir et al. “A comprehensive review of deep learning-based single image super-resolution.” PeerJ Comput Sci. Jul. 13, 2021;7:e621. doi: 10.7717/peerj-cs.621. PMID: 34322592; PMCID: PMC8293932. (Year: 2021). [cited by examiner]
Wang et al., “Deep learning for image super-resolution: A survey.” IEEE transactions on pattern analysis and machine intelligence 43, No. 10 (2020): 3365-3387. (Year: 2020). [cited by examiner]
Michelini et al., “Multi-Grid Back-Projection Networks,” in IEEE Journal of Selected Topics in Signal Processing, vol. 15, No. 2, pp. 279-294, Feb. 2021 (Year: 2021). [cited by examiner]
Pan et al., “Super-Resolution of Single Remote Sensing Image Based on Residual Dense Backprojection Networks,” in IEEE Transactions on Geoscience and Remote Sensing, vol. 57, No. 10, pp. 7918-7933, Oct. 2019 (Year: 2019… [cited by examiner]
Y. Mei et al., “Pyramid Attention Networks for Image Restoration,” arXiv:2004.13824v4, Jun. 3, 2020, 19 pages. [cited by applicant]
J. Kim et al., “Accurate Image Super-Resolution Using Very Deep Convolutional Networks,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), arXiv:1511.04587v2, Nov. 11, 2016, 9 pages. [cited by applicant]
W.-S. Lai et al., “Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), arXiv:1704.03915v2, Oct. 9, 2017, 9 pages. [cited by applicant]
A. Vaswani et al., “Attention Is All You Need,” Advances in Neural Information Processing Systems, arXiv:1706.03762v5, Dec. 6, 2017, 15 pages. [cited by applicant]
B. Lim et al., “Enhanced Deep Residual Networks for Single Image Super-Resolution,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), arXiv:1707.02921v1, Jul. 10, 2017, 9 pages. [cited by applicant]
Y. Zhang et al., “Image Super-Resolution Using Very Deep Residual Channel Attention Networks,” European Conference on Computer Vision, arXiv:1807.02758v2, Jul. 12, 2018, 16 pages. [cited by applicant]
A. Buades et al., “A Non-local Algorithm for Image Denoising,” 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), vol. 2, Jun. 2005, pp. 60-65. [cited by applicant]
M. Haris et al., “Deep Back-Projection Networks for Single Image Super-resolution,” IEEE Transactions on Pattern Analysis and Machine Intelligence, arXiv:1904.05677v2, Jun. 13, 2020, 14 pages. [cited by applicant]
Z.-S. Liu et al., “Hierarchical Back Projection Network for Image Super-Resolution,” IEEE/CVF Conference on Computer Vision and Pattern Recognition, arXiv:1906.06874v2, Jun. 20, 2019, 10 pages. [cited by applicant]
Z.- S. Liu et al., “Image Super-Resolution via Attention based Back Projection Networks,” IEEE/CVF Conference on Computer Vision and Pattern Recognition, arXiv:1910.04476v, Oct. 10, 2019, 9 pages. [cited by applicant]
T. Dai et al., “Second-order Attention Network for Single Image Super-Resolution,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Jun. 1, 2019, 10 pages. [cited by applicant]
N. Parmar et al, “Image Transformer,” International Conference on Machine Learning, arXiv:1802.05751v3, Jun. 15, 2018, 10 pages. [cited by applicant]
X. Wang et al., “Non-local Neural Networks,” Conference on Computer Vision and Pattern Recognition, arXiv:1711.07971v3, Apr. 13, 2018, 10 pages. [cited by applicant]
Cited By (1)
US 12,694,574