IP Library › Granted Patent US 12,307,635
Granted Patent B2
US 12,307,635 · App. 17/746,769 · Granted May 20, 2025

Image signal processor

Inventors: Yuri Dolgin (Haifa, IL); Maksym Aslianskyi (Tel Aviv, IL); Costia Parfenyev (Haifa, IL); Eran Pinhasov (Zichron Yaakov, IL); Victor Pinto (Zichron Yaakov, IL)
Assignee: QUALCOMM Incorporated
G06T5/70G06T3/40G06T2207/10016G06T2207/20081G06T2207/20182
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,307,635
App. No.
17/746,769
Granted
May 20, 2025
Kind
B2
Abstract

The present disclosure generally relates to image processing. For example, aspects of the present disclosure include systems and techniques for performing spatial and temporal processing of image data. Certain aspects provide an apparatus for processing frame data. The apparatus generally includes a memory, and one or more processors coupled to the memory, the one or more processors configured to: perform a first noise reduction operation based on first frame data via a machine learning component to generate first processed frame data; generate first feedback data based on the first processed frame data; and perform, via the machine learning component, a second noise reduction operation based on second frame data and the first feedback data.

Claims (53)

1. An apparatus for processing frame data, the apparatus comprising:

a memory; and

one or more processors coupled to the memory, the one or more processors configured to:

perform a first noise reduction operation based on first frame data via a machine learning component to generate first processed frame data;

generate first feedback data based on the first processed frame data;

perform, via the machine learning component, a second noise reduction operation based on second frame data and the first feedback data; and

perform, via a frame processing component, a third noise reduction operation based on the first processed frame data and third frame data to generate second processed frame data.

2. The apparatus of claim 1 , wherein, to generate the first feedback data, the one or more processors are configured to warp one or more pixels of the first processed frame data.

3. The apparatus of claim 2 , wherein the one or more processors are configured to warp the one or more pixels to align one or more features of the first processed frame data with one or more features of the second frame data.

4. The apparatus of claim 1 , wherein the one or more processors are further configured to filter the second frame data via an image filter to generate filtered frame data, and wherein the second noise reduction operation is performed further based on the filtered frame data.

5. The apparatus of claim 4 , wherein the one or more processors are configured to filter the second frame data via the image filter based on the first feedback data.

6. The apparatus of claim 1 , wherein the one or more processors are further configured to generate, via an image filter, filtered frame data based on the first frame data, wherein the second noise reduction operation being performed based on the first processed frame data comprises the second noise reduction operation being performed based on the filtered frame data.

7. The apparatus of claim 1 , wherein the first frame data comprises a down-sampled version of the third frame data.

8. The apparatus of claim 1 , wherein the third noise reduction operation is performed using machine learning.

9. The apparatus of claim 1 , wherein the one or more processors are further configured to:

generate second feedback data based on the second processed frame data; and

perform a fourth noise reduction operation based on fourth frame data and the second feedback data, wherein the second frame data comprises a down-sampled version of the fourth frame data.

10. The apparatus of claim 1 , wherein the one or more processors are further configured to generate, via an image filter, a filtered output frame based on the second processed frame data and the third frame data.

11. The apparatus of claim 1 , wherein the one or more processors are configured to perform the second noise reduction operation to generate a first output frame of a video, and wherein the one or more processors are further configured to:

perform the third noise reduction operation based on the second frame data to generate second processed frame data;

generate second feedback data based on the second processed frame data; and

perform a fourth noise reduction operation based on third frame data and the second feedback data to generate a second output frame of the video.

12. The apparatus of claim 1 , wherein:

the first frame data is associated with a first patch of a frame; and

the one or more processors are further configured to perform a third noise reduction operation based on a second patch of the frame and the first processed frame data generated using the first patch of the frame.

13. A method for processing frame data, the method comprising:

performing a first noise reduction operation based on first frame data via a machine learning component to generate first processed frame data;

generating first feedback data based on the first processed frame data;

performing, via the machine learning component, a second noise reduction operation based on second frame data and the first feedback data; and

performing, via a frame processing component, a third noise reduction operation based on the first processed frame data and third frame data to generate second processed frame data.

14. The method of claim 13 , wherein generating the first feedback data comprises warping one or more pixels of the first processed frame data.

15. The method of claim 14 , wherein the one or more pixels are warped to align one or more features of the first processed frame data with one or more features of the second frame data.

16. The method of claim 13 , further comprising filtering the second frame data via an image filter to generate filtered frame data, and wherein the second noise reduction operation is performed further based on the filtered frame data.

17. The method of claim 16 , wherein the second frame data is filtered via the image filter based on the first feedback data.

18. The method of claim 13 , further comprising generating, via an image filter, filtered frame data based on the first frame data, wherein the second noise reduction operation being performed based on the first processed frame data comprises the second noise reduction operation being performed based on the filtered frame data.

19. The method of claim 13 , wherein the first frame data comprises a down-sampled version of the third frame data.

20. The method of claim 13 , wherein the third noise reduction operation is performed using machine learning.

21. The method of claim 13 , wherein the method further comprises:

generating second feedback data based on the second processed frame data; and

performing a fourth noise reduction operation based on fourth frame data and the second feedback data, wherein the second frame data comprises a down-sampled version of the fourth frame data.

22. The method of claim 13 , wherein method further comprises generating, via an image filter, a filtered output frame based on the second processed frame data and the third frame data.

23. The method of claim 13 , wherein the second noise reduction operation is performed to generate a first output frame of a video, and wherein the method further comprises:

perform the third noise reduction operation based on the second frame data to generate second processed frame data;

generate second feedback data based on the second processed frame data; and

perform a fourth noise reduction operation based on third frame data and the second feedback data to generate a second output frame of the video.

24. The method of claim 13 , wherein:

the first frame data is associated with a first patch of a frame; and

the method further comprises performing a third noise reduction operation based on a second patch of the frame and the first processed frame data generated using the first patch of the frame.

25. A non-transitory computer-readable medium having instructions stored thereon, that when executed by a processor, causes the processor to:

perform a first noise reduction operation based on first frame data via a machine learning component to generate first processed frame data;

generate first feedback data based on the first processed frame data;

perform, via the machine learning component, a second noise reduction operation based on second frame data and the first feedback data; and

perform, via a frame processing component, a third noise reduction operation based on the first processed frame data and third frame data to generate second processed frame data.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 22, 2022
From: DOLGIN, YURI; ASLIANSKYI, MAKSYM; PARFENYEV, COSTIA; PINHASOV, ERAN; PINTO, VICTOR
To: QUALCOMM INCORPORATED
Reel/Frame 060599/0853 →
Continuity (1)
Related Publication 20230377096A1 · Nov 23, 2023
References Cited (12)
US 20180114294A1 · Yang · 2018 [cited by examiner]
Lee, Denoising Algorithm for CFA Image Sensors Considering Inter-Channel Correlation, Sensors 2017, 17, 1236 (Year: 2017). [cited by examiner]
Hasselgren, Neural Temporal Adaptive Sampling and Denoising, EUROGRAPHICS 2020, DOI: 10.1111/cgf. 13919, vol. 39 ( 2020), No. 2 (Year: 2020). [cited by examiner]
Chen, Deep RNNs for Video Denoising, Proc. of SPIE vol. 9971 99711T-1, doi: 10.1117/12.2239260, 2016 (Year: 2016). [cited by examiner]
Chen X., et al., “Deep RNNs for Video Denoising”, [Proceedings of SPIE ISSN 0277-786X vol. 10524], SPIE, US, vol. 9971, Sep. 28, 2016, pp. 99711T-1-99711T-10, XP060078021, DOI: 10.1117/12.2239260, ISBN: 978-1-5106-1533-… [cited by applicant]
Hasselgren J., et al., “Neural Temporal Adaptive Sampling and Denoising”, Computer Graphics Forum : Journal of the European Association for Computer Graphics, Wiley-Blackwell, Oxford, vol. 39, No. 2, Jul. 13, 2020, pp. … [cited by applicant]
International Search Report and Written Opinion—PCT/US2023/066777—ISA/EPO—Jul. 24, 2023. [cited by applicant]
Lee M.S., et al., “Denoising Algorithm for CFA Image Sensors Considering Inter-Channel Correlation”, Sensors, vol. 17, No. 6, May 28, 2017, pp. 1-22, XP55590892, DOI: 10.3390/s17061236, Figure 3. [cited by applicant]
Liu Z., et al., “Image Denoising Based on A Cnn Model”, 2018 4th International Conference on Control, Automation and Robotics (ICCAR), IEEE, Apr. 20, 2018, pp. 389-393, XP033358899, DOI: 10.1109/ICCAR.2018.8384706 [retr… [cited by applicant]
Ma B., et al., “A Multi-Scale Video Denoising Algorithm for Raw Image”, Methods, Algorithms and Applications, Sep. 5, 2022, 14 Pages, XP93063830, [retrieved on Jul. 13, 2023], Figure 1, Section “4.3 Denoising Module”. [cited by applicant]
Sun L., et al., “Deep Maximum a Posterior Estimator for Video Denoising”, arxiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, vol. 129, No. 10, Aug. 4, 2021, pp. 2827-2845, XP037… [cited by applicant]
Yan Z., et al., “On Combining CNN With Non-Local Self-Similarity Based Image Denoising Methods”, IEEE Access, IEEE, USA, vol. 8, Dec. 30, 2019, pp. 14789-14797, XP011768334, DOI: 10.1109/ACCESS.2019.2962809, [retrieved … [cited by applicant]