IP Library › Granted Patent US 12,400,294
Granted Patent B2
US 12,400,294 · App. 17/823,960 · Granted Aug 26, 2025

Image processing device and super-resolution processing method

Inventors: Yung-Hui Li (New Taipei, TW); Chi-En Huang (New Taipei, TW)
Assignees: HON HAI PRECISION INDUSTRY CO., LTD.; Foxconn Technology Group Co., Ltd.
G06T3/4053G06T3/4046G06T3/4076
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,400,294
App. No.
17/823,960
Granted
Aug 26, 2025
Kind
B2
Abstract

An image processing device is provided, which includes an image capture circuit and a processor. The image capturing circuit is configured for capturing a high-resolution image. The processor is connected to the image capturing circuit, and performing a super-resolution model and an attention model, where the processor is configured to perform following operations for: performing down sampling processing on the high-resolution image to generate a low-resolution image; performing super-resolution processing on the low-resolution image using the super-resolution model to generate a super-resolution image; applying the attention model to the high-resolution image and the super-resolution image to generate an attention weighted high-resolution image and an attention weighted super-resolution image, and calculating a first loss according to the attention weighted high-resolution image and the attention weighted super-resolution image, thereby updating the super-resolution model.

Claims (52)

1. An image processing device, comprising:

an image capturing circuit, configured for capturing a high-resolution image; and

a processor, connected to the image capturing circuit, and performing a super-resolution model and an attention model, wherein the processor is configured to perform following operations for:

performing down sampling processing on the high-resolution image to generate a low-resolution image;

performing super-resolution processing on the low-resolution image using the super-resolution model to generate a super-resolution image;

applying the attention model to the high-resolution image and the super-resolution image to generate an attention weighted high-resolution image and an attention weighted super-resolution image, and calculating a first loss according to the attention weighted high-resolution image and the attention weighted super-resolution image; and

updating the super-resolution model according to the first loss.

2. The image processing device of claim 1 , wherein the processor is further configured to perform following operations:

applying the attention model to the high-resolution image and the super-resolution image for performing image filtering processing to generate the attention weighted high-resolution image and the attention weighted super-resolution image.

3. The image processing device of claim 2 , wherein the image filtering processing is an element-wise multiplication operation, an element-wise addition operation, an element-wise subtraction operation, or a combination thereof.

4. The image processing device of claim 1 , wherein the attention model comprises a mask processing model corresponding to at least one image mask, and the processor is further configured to perform following operations:

performing image filtering processing on the high-resolution image and the super-resolution image respectively using the mask processing model to generate the attention weighted high-resolution image and the attention weighted super-resolution image.

5. The image processing device of claim 4 , wherein the at least one image mask corresponds to at least one image region of interest, and the processor is further configured to perform following operations:

generate the attention weighted high-resolution image and the attention weighted super-resolution image according to the at least one image region of interest, the high-resolution image and the super-resolution image.

6. The image processing device of claim 5 , wherein the processor is further configured to perform following operations:

performing an element-wise multiplication operation on the high-resolution image and the super-resolution image respectively according to the at least one image region of interest to generate the attention weighted high-resolution image and the attention weighted super-resolution image, wherein the attention weighted high-resolution image and the attention weighted super-resolution image correspond to the high-resolution image and the super-resolution image, respectively; and

performing a distance function operation according to the attention weighted high-resolution image and the attention weighted super-resolution image to generate the first loss, wherein the distance function operation is an L1 distance operation, an L2 distance operation, a Charbonnier distance operation, or a combination thereof.

7. The image processing device of claim 4 , wherein the at least one image mask corresponds to at least one weight set, and the processor is further configured to perform following operations:

generating the attention weighted high-resolution image and the attention weighted super-resolution image according to the at least one weight set, the high-resolution image and the super-resolution image.

8. The image processing device of claim 7 , wherein the processor is further configured to perform following operations:

performing an element-wise multiplication operation on the high-resolution image and the super-resolution image respectively according to the at least one weight set to generate the attention weighted high-resolution image and the attention weighted super-resolution image, wherein the attention weighted high-resolution image and the attention weighted super-resolution image correspond to the high-resolution image and the super-resolution image, respectively; and

performing a distance function operation according to the attention weighted high-resolution image and the attention weighted super-resolution image to generate the first loss, wherein the distance function operation is an L1 distance operation, an L2 distance operation, a Charbonnier distance operation, or a combination thereof.

9. The image processing device of claim 1 , wherein the processor is further configured to perform following operations:

calculating a second loss according to the high-resolution image and the super-resolution image using the super-resolution model.

10. The image processing device of claim 9 , wherein the processor is further configured to perform following operations:

calculating a total loss according to the first loss and the second loss, and performing a back propagation operation on the super-resolution model according to the total loss to generate an updated super-resolution model.

11. A super-resolution processing method, comprising:

capturing a high-resolution image, and performing down sampling processing on the high-resolution image to generate a low-resolution image;

performing super-resolution processing on the low-resolution image using a super-resolution model to generate a super-resolution image;

applying an attention model to the high-resolution image and the super-resolution image to generate an attention weighted high-resolution image and an attention weighted super-resolution image, and calculating a first loss according to the attention weighted high-resolution image and the attention weighted super-resolution image; and

updating the super-resolution model according to the first loss.

12. The super-resolution processing method of claim 11 , wherein the step of applying the attention model to the high-resolution image and the super-resolution image to generate the attention weighted high-resolution image and the attention weighted super-resolution image comprises:

applying the attention model to the high-resolution image and the super-resolution image for performing image filtering processing to generate the attention weighted high-resolution image and the attention weighted super-resolution image.

13. The super-resolution processing method of claim 12 , wherein the image filtering processing is an element-wise multiplication operation, an element-wise addition operation, an element-wise subtraction operation, or a combination thereof.

14. The super-resolution processing method of claim 11 , wherein the attention model comprises a mask processing model corresponding to at least one image mask, and the step of applying the attention model to the high-resolution image and the super-resolution image to generate the attention weighted high-resolution image and the attention weighted super-resolution image comprises:

performing image filtering processing on the high-resolution image and the super-resolution image respectively using the mask processing model to generate the attention weighted high-resolution image and the attention weighted super-resolution image.

15. The super-resolution processing method of claim 14 , wherein the at least one image mask corresponds to at least one image region of interest, and the step of applying the attention model to the high-resolution image and the super-resolution image to generate the attention weighted high-resolution image and the attention weighted super-resolution image further comprises:

generate the attention weighted high-resolution image and the attention weighted super-resolution image according to the at least one image region of interest, the high-resolution image and the super-resolution image.

16. The super-resolution processing method of claim 15 , wherein the step of applying the attention model to the high-resolution image and the super-resolution image to generate the attention weighted high-resolution image and the attention weighted super-resolution image further comprises:

performing an element-wise multiplication operation on the high-resolution image and the super-resolution image respectively according to the at least one image region of interest to generate the attention weighted high-resolution image and the attention weighted super-resolution image, wherein the attention weighted high-resolution image and the attention weighted super-resolution image correspond to the high-resolution image and the super-resolution image, respectively,

wherein the step of calculating the first loss according to the attention weighted high-resolution image and the attention weighted super-resolution image comprises:

performing a distance function operation according to the attention weighted high-resolution image and the attention weighted super-resolution image to generate the first loss, wherein the distance function operation is an L1 distance operation, an L2 distance operation, a Charbonnier distance operation, or a combination thereof.

17. The super-resolution processing method of claim 14 , wherein the at least one image mask corresponds to at least one weight set, and the step of applying the attention model to the high-resolution image and the super-resolution image to generate the attention weighted high-resolution image and the attention weighted super-resolution image further comprises:

generate the attention weighted high-resolution image and the attention weighted super-resolution image according to the at least one weight set, the high-resolution image and the super-resolution image.

18. The super-resolution processing method of claim 17 , wherein the step of applying the attention model to the high-resolution image and the super-resolution image to generate the attention weighted high-resolution image and the attention weighted super-resolution image further comprises:

performing an element-wise multiplication operation on the high-resolution image and the super-resolution image respectively according to the at least one weight set to generate the attention weighted high-resolution image and the attention weighted super-resolution image, wherein the attention weighted high-resolution image and the attention weighted super-resolution image correspond to the high-resolution image and the super-resolution image, respectively,

wherein the step of calculating the first loss according to the attention weighted high-resolution image and the attention weighted super-resolution image comprises:

performing a distance function operation according to the attention weighted high-resolution image and the attention weighted super-resolution image to generate the first loss, wherein the distance function operation is an L1 distance operation, an L2 distance operation, a Charbonnier distance operation, or a combination thereof.

19. The super-resolution processing method of claim 11 , wherein the step of updating the super-resolution model according to the first loss comprises:

calculating a second loss according to the high-resolution image and the super-resolution image using the super-resolution model.

20. The super-resolution processing method of claim 19 , wherein the step of updating the super-resolution model according to the first loss further comprises:

calculating a total loss according to the first loss and the second loss, and performing a back propagation operation on the super-resolution model according to the total loss to generate an updated super-resolution model.

Assignments (2)
CHANGE OF NAME Recorded Jul 10, 2025
From: HONGFUJIN PRECISION INDUSTRY (SHENZHEN) CO., LTD.
To: FOXCONN TECHNOLOGY GROUP CO., LTD.
Reel/Frame 072406/0386 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 1, 2022
From: LI, YUNG-HUI; HUANG, CHI-EN
To: HON HAI PRECISION INDUSTRY CO., LTD.; HONGFUJIN PRECISION INDUSTRY (SHENZHEN) CO., LTD.
Reel/Frame 060972/0259 →
Continuity (2)
Provisional Application 63239423 · Sep 1, 2021
Related Publication 20230063201A1 · Mar 2, 2023
References Cited (30)
US 20130011078A1 · Phan et al. · 2013 [cited by applicant]
US 20160171658A1 · Matson et al. · 2016 [cited by applicant]
US 20200311871A1 · Yu et al. · 2020 [cited by applicant]
US 20210004935A1 · Yao et al. · 2021 [cited by applicant]
US 20210073945A1 · Kim et al. · 2021 [cited by applicant]
US 20210104018A1 · Moon et al. · 2021 [cited by applicant]
US 20210133925A1 · Lee et al. · 2021 [cited by applicant]
US 20220108212A1 · Zhai · 2022 [cited by examiner]
US 20220286696A1 · Gao · 2022 [cited by examiner]
US 20240185386A1 · Pan · 2024 [cited by examiner]
CN 103095977A · 2013 [cited by applicant]
CN 109034198A · 2018 [cited by applicant]
CN 109816593A · 2019 [cited by applicant]
CN 110175953A · 2019 [cited by applicant]
CN 110717856A · 2020 [cited by applicant]
CN 110852948A · 2020 [cited by applicant]
CN 111222466A · 2020 [cited by applicant]
CN 111402137A · 2020 [cited by applicant]
TW I419059B · 2013 [cited by applicant]
WO 2020187220A1 · 2020 [cited by applicant]
WO 2020238558A1 · 2020 [cited by applicant]
C. Chen, D. Gong, H. Wang, Z. Li and K.-Y. K. Wong, “Learning Spatial Attention for Face Super-Resolution,” in IEEE Transactions on Image Processing, vol. 30, pp. 1219-1231, 2021, doi: 10.1109/TIP.2020.3043093. Date of … [cited by examiner]
J. Wang, et al., “Lightweight Feedback Convolution Neural Network for Remote Sensing Images Super-Resolution,” in IEEE Access, vol. 9, pp. 15992-16003 (Year: 2021). [cited by applicant]
T. Dai, et al., “Image Super-Resolution via Residual Block Attention Networks”, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Korea (South), pp. 3879-3886 (Year: 2019). [cited by applicant]
Fan Fan, et al., “Abdominal MRI image multi-scale super-resolution reconstruction based on parallel channel-spatial attention mechanism”, Journal of Computer Applications, Dec. 10, 2020, p. 3624-3630, vol. 40, No. 12, C… [cited by applicant]
Jing Chen, “Research on Image Super-resolution Reconstruction Technology Based on Deep Learning”, Master's degree thesis, Apr. 2020, Nanjing University of Posts and Telecommunications, China. [cited by applicant]
Da li Gao, “Research on Airport Runway FOD Detection Algorithm Based on Convolutional Neural Network”, Matster's Degree thesis, Jul. 2020, Xidian University, China. [cited by applicant]
Yueqi Zhong, “Principles, Techniques and Applications of Artificial Intelligence”, Sep. 2020, p. 113, Donghua University Press Co., Ltd., China. [cited by applicant]
Zhihao Fan, et al., “Mask Attention Networks Rethinking and Strengthen Transformer”, ARXIV, May 25, 2021, p. 1-10, Cornell University, United States. (https://doi.org/10.48550/arXiv.2103.13597). [cited by applicant]
Gaopeng Hu, et al., “Image super-resolution reconstruction based on deep progressive back-projection attention network”, Journal of Computer Applications, Jul. 10, 2020, p. 2077-2083, vol. 40, No. 7, China Academic Jour… [cited by applicant]