IP Library › Granted Patent US 11,363,213
Granted Patent B1
US 11,363,213 · App. 17/187,357 · Granted Jun 14, 2022

Minimizing ghosting in high dynamic range image processing

Inventors: Ki Sun Song (Seoul, KR); Yi-Chun Lu (San Diego, CA); Shang-Chih Chuang (San Diego, CA); Xiaoyun Jiang (San Diego, CA); Nyeongkyu Kwon (San Diego, CA)
Assignee: QUALCOMM Incorporated
H04N5/2355G06T7/20G06T2207/10144G06T2207/20221
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,363,213
App. No.
17/187,357
Granted
Jun 14, 2022
Kind
B1
Abstract

Methods, systems, and devices for minimizing ghosting in high dynamic range image processing are described. The method may include capturing from a sensor of the device a downscaled first frame of a first exposure length and a downscaled second frame of a second exposure length that is longer than the first exposure length, identifying a highlight region associated with the downscaled first frame and a motion region associated with the downscaled first frame and with the downscaled second frame, blending the motion region in accordance with determining whether at least a portion of the motion region overlaps the highlight region, and incorporating the blending of the motion region in a set of full resolution frames.

Claims (91)

1. A method for image processing at a device, comprising:

capturing from a sensor of the device a downscaled first frame of a first exposure length and a downscaled second frame of a second exposure length that is longer than the first exposure length;

identifying a highlight region associated with the downscaled first frame and a motion region associated with the downscaled first frame and with the downscaled second frame;

blending the motion region in accordance with determining whether at least a portion of the motion region overlaps the highlight region; and

incorporating the blending of the motion region in a set of full resolution frames.

2. The method of claim 1 , further comprising:

generating a highlight map comprising one or more highlight regions identified in the downscaled first frame, wherein the one or more highlight regions include the highlight region; and

generating a motion map comprising one or more motion regions identified in the downscaled first frame and in the downscaled second frame, wherein the one or more motion regions include the motion region.

3. The method of claim 2 , further comprising:

analyzing coordinates of the one or more highlight regions in relation to coordinates of the one or more motion regions; and

determining, in accordance with the analyzing, that at least the portion of the motion region overlaps the highlight region.

4. The method of claim 3 , further comprising:

generating a refined highlight map by adding at least the portion of the motion region that overlaps the highlight region to the highlight map; and

generating a refined motion map by removing at least the portion of the motion region that overlaps the highlight region from the motion map.

5. The method of claim 4 , wherein the set of full resolution frames comprises a first full resolution frame captured at the first exposure length and a second full resolution frame captured at the second exposure length.

6. The method of claim 5 , further comprising:

detecting a second highlight region of the first full resolution frame based at least in part on the refined highlight map; and

blending the detected highlight region of the first full resolution frame into a portion of the first full resolution frame that surrounds the detected highlight region.

7. The method of claim 6 , further comprising:

detecting a second motion region of the second full resolution frame based at least in part on the refined motion map; and

blending the detected motion region of the second full resolution frame into a portion of the second full resolution frame that surrounds the detected motion region.

8. The method of claim 7 , further comprising:

fusing, after the blending of the detected highlight region and the detected motion region, the first full resolution frame with the second full resolution frame.

9. The method of claim 1 , wherein capturing the downscaled first frame and the downscaled second frame comprises:

capturing the downscaled first frame at a first resolution that is less than a full resolution of the sensor; and

capturing the downscaled second frame at a second resolution that is the same or less than the first resolution.

10. The method of claim 9 , further comprising:

capturing the set of full resolution frames at the full resolution of the sensor.

11. An apparatus for image processing at a device, comprising:

a processor;

memory coupled with the processor; and

instructions stored in the memory and executable by the processor to cause the apparatus to:

capture from a sensor of the device a downscaled first frame of a first exposure length and a downscaled second frame of a second exposure length that is longer than the first exposure length;

identify a highlight region associated with the downscaled first frame and a motion region associated with the downscaled first frame and with the downscaled second frame;

blend the motion region in accordance with determining whether at least a portion of the motion region overlaps the highlight region; and

incorporate the blending of the motion region in a set of full resolution frames.

12. The apparatus of claim 11 , wherein the instructions are further executable by the processor to cause the apparatus to:

generate a highlight map comprising one or more highlight regions identified in the downscaled first frame, wherein the one or more highlight regions include the highlight region; and

generate a motion map comprising one or more motion regions identified in the downscaled first frame and in the downscaled second frame, wherein the one or more motion regions include the motion region.

13. The apparatus of claim 12 , wherein the instructions are further executable by the processor to cause the apparatus to:

analyze coordinates of the one or more highlight regions in relation to coordinates of the one or more motion regions; and

determine, in accordance with the analyzing, that at least the portion of the motion region overlaps the highlight region.

14. The apparatus of claim 13 , wherein the instructions are further executable by the processor to cause the apparatus to:

generate a refined highlight map by adding at least the portion of the motion region that overlaps the highlight region to the highlight map; and

generate a refined motion map by removing at least the portion of the motion region that overlaps the highlight region from the motion map.

15. The apparatus of claim 14 , wherein the set of full resolution frames comprises a first full resolution frame captured at the first exposure length and a second full resolution frame captured at the second exposure length.

16. The apparatus of claim 15 , wherein the instructions are further executable by the processor to cause the apparatus to:

detect a second highlight region of the first full resolution frame based at least in part on the refined highlight map; and

blend the detected highlight region of the first full resolution frame into a portion of the first full resolution frame that surrounds the detected highlight region.

17. The apparatus of claim 16 , wherein the instructions are further executable by the processor to cause the apparatus to:

detect a second motion region of the second full resolution frame based at least in part on the refined motion map; and

blend the detected motion region of the second full resolution frame into a portion of the second full resolution frame that surrounds the detected motion region.

18. The apparatus of claim 17 , wherein the instructions are further executable by the processor to cause the apparatus to:

fuse, after the blending of the detected highlight region and the detected motion region, the first full resolution frame with the second full resolution frame.

19. An apparatus for image processing at a device, comprising:

means for capturing from a sensor of the device a downscaled first frame of a first exposure length and a downscaled second frame of a second exposure length that is longer than the first exposure length;

means for identifying a highlight region associated with the downscaled first frame and a motion region associated with the downscaled first frame and with the downscaled second frame;

means for blending the motion region in accordance with determining whether at least a portion of the motion region overlaps the highlight region; and

means for incorporating the blending of the motion region in a set of full resolution frames.

20. The apparatus of claim 19 , further comprising:

means for generating a highlight map comprising one or more highlight regions identified in the downscaled first frame, wherein the one or more highlight regions include the highlight region; and

means for generating a motion map comprising one or more motion regions identified in the downscaled first frame and in the downscaled second frame, wherein the one or more motion regions include the motion region.

21. A method for image processing at a device, comprising:

capturing from a sensor of the device a downscaled first frame of a first exposure length and a downscaled second frame of a second exposure length that is longer than the first exposure length;

identifying a highlight region associated with the downscaled first frame and a motion region associated with the downscaled first frame and with the downscaled second frame;

blending the motion region in accordance with determining whether at least a portion of the motion region overlaps the highlight region; and

incorporating the blending of the motion region in a set of full resolution frames.

22. The method of claim 21 , further comprising:

generating a highlight map comprising one or more highlight regions identified in the downscaled first frame, wherein the one or more highlight regions include the highlight region; and

generating a motion map comprising one or more motion regions identified in the downscaled first frame and in the downscaled second frame, wherein the one or more motion regions include the motion region.

23. The method of claim 22 , further comprising:

analyzing coordinates of the one or more highlight regions in relation to coordinates of the one or more motion regions; and

determining, in accordance with the analyzing, that at least the portion of the motion region overlaps the highlight region.

24. The method of claim 23 , further comprising:

generating a refined highlight map by adding at least the portion of the motion region that overlaps the highlight region to the highlight map; and

generating a refined motion map by removing at least the portion of the motion region that overlaps the highlight region from the motion map.

25. The method of claim 24 , wherein:

the set of full resolution frames comprises a first full resolution frame captured at the first exposure length and a second full resolution frame captured at the second exposure length.

26. The method of claim 25 , further comprising:

detecting a second highlight region of the first full resolution frame based at least in part on the refined highlight map; and

blending the detected highlight region of the first full resolution frame into a portion of the first full resolution frame that surrounds the detected highlight region.

27. The method of claim 26 , further comprising:

detecting a second motion region of the second full resolution frame based at least in part on the refined motion map; and

blending the detected motion region of the second full resolution frame into a portion of the second full resolution frame that surrounds the detected motion region.

28. The method of claim 27 , further comprising:

fusing, after the blending of the detected highlight region and the detected motion region, the first full resolution frame with the second full resolution frame.

29. The method of any of claims 21 through 28 , wherein capturing the downscaled first frame and the downscaled second frame comprises:

capturing the downscaled first frame at a first resolution that is less than a full resolution of the sensor; and

capturing the downscaled second frame at a second resolution that is the same or less than the first resolution.

30. The method of claim 29 , further comprising:

capturing the set of full resolution frames at the full resolution of the sensor.

Assignments (2)
CORRECTIVE ASSIGNMENT TO CORRECT THE REPLACING ASSIGNMENT DOCUMENT PREVIOUSLY RECORDED AT REEL: 056218 FRAME: 0527. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT . Recorded Sep 24, 2021
From: SONG, KI SUN; LU, YI-CHUN; CHUANG, SHANG-CHIH; JIANG, XIAOYUN; KWON, NYEONGKYU
To: QUALCOMM INCORPORATED
Reel/Frame 057598/0876 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 12, 2021
From: SONG, KI SUN; LU, YI-CHUN; CHUANG, SHANG-CHIH; JIANG, XIAOYUN
To: QUALCOMM INCORPORATED
Reel/Frame 056218/0527 →
Cited By (1)
US 12,568,314