IP Library › Granted Patent US 11,184,553
Granted Patent B1
US 11,184,553 · App. 17/029,526 · Granted Nov 23, 2021

Image signal processing in multi-camera system

Inventors: Tai-Hsin Liu (Taipei, TW); Wei-Chih Liu (Taipei, TW); Wen-Chun Feng (Taipei, TW)
Assignee: QUALCOMM Incorporated
H04N5/2355H04N5/2258H04N5/2621H04N5/2628
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,184,553
App. No.
17/029,526
Granted
Nov 23, 2021
Kind
B1
Abstract

A processor generates a depth map from two images, including a high-dynamic range (HDR) image. The processor receives, from a first image sensor, one or more first images, wherein the one or more first images have a first field-of-view (FOV) and a first resolution, and receives, from a second image sensor, one or more second images, wherein the one or more second images have a second FOV and a second resolution. The processor generates a first HDR image from the one or more first images, and performs tone alignment on the one or more second images based on the first HDR image to produce a second HDR image. The processor generates a depth map using the first HDR image and the second HDR image. The processor may use the depth map to apply a bokeh effect to the first HDR image.

Claims (74)

1. A device configured for camera processing, the device comprising:

a memory configured to receive images; and

one or more processors in communication with the memory, the one or more processors configured to:

receive, from a first image sensor, one or more first images, wherein the one or more first images have a first field-of-view (FOV) and a first resolution;

generate a first high-dynamic range (HDR) image from the one or more first images;

receive, from a second image sensor, one or more second images, wherein the one or more second images have a second FOV and a second resolution;

generate a second high-dynamic range (HDR) image from the one or more second images, including performing tone alignment on the one or more second images based on the first HDR image; and

generate a depth map using the first HDR image and the second HDR image.

2. The device of claim 1 , wherein the one or more processors are further configured to:

synchronize the second FOV of the one or more second images to the first FOV of the one or more first images; and

synchronize the second resolution of the one or more second images to the first resolution of the one or more first images, and

wherein the one or more processors generate the second HDR image from the one or more second images after the synchronization of the second FOV and the synchronization of the second resolution.

3. The device of claim 2 , wherein to synchronize the second FOV of the one or more second images to the first FOV of the one or more first images, the one or more processors are further configured to:

cause the second image sensor to crop the second FOV to match the first FOV.

4. The device of claim 3 , wherein to synchronize the second FOV of the one or more second images to the first FOV of the one or more first images further, the one or more processors are further configured to:

spatially align the second FOV to match the first FOV.

5. The device of claim 2 , wherein to synchronize the second resolution of the one or more second images to the first resolution of the one or more first images, the one or more processors are further configured to:

cause the second image sensor to perform remosaicing on the one or more second images having the synchronized second FOV.

6. The device of claim 2 , wherein to synchronize the second resolution of the one or more second images to the first resolution of the one or more first images, the one or more processors are further configured to:

cause the second image sensor to perform a binning operation on the one or more second images having the synchronized second FOV.

7. The device of claim 1 , wherein to generate the first HDR image from the one or more first images, the one or more processors are further configured to generate the first HDR image using a plurality of the one or more first images using a multi-frame HDR technique, and

wherein to generate the second HDR image from the one or more second images, the one or more processors are further configured to generate the second HDR image using a plurality of the one or more second images using the multi-frame HDR technique.

8. The device of claim 1 , wherein to generate the first HDR image from the one or more first images, the one or more processors are further configured to generate the first HDR image using a plurality of the one or more first images using a first HDR technique, and

wherein to generate the second HDR image from the one or more second images, the one or more processors are further configured to generate the second HDR image using a second HDR technique based on a tone range produced by the first HDR technique.

9. The device of claim 8 , wherein the first HDR technique is a multi-frame HDR technique and wherein the second HDR technique is one of a three-exposure HDR technique or an adaptive dynamic range coding (ADRC) technique.

10. The device of claim 8 , wherein the first HDR technique and the second HDR technique are different techniques.

11. The device of claim 10 , wherein the second HDR technique is an adaptive dynamic range coding (ADRC) technique.

12. The device of claim 1 , wherein the one or more processors are further configured to:

apply an image effect to the first HDR image using the depth map.

13. The device of claim 12 , wherein the image effect is a bokeh effect.

14. The device of claim 1 , further comprising:

the first image sensor;

the second image sensor; and

a display configured to display the first HDR image.

15. A method of camera processing, the method comprising:

receiving, from a first image sensor, one or more first images, wherein the one or more first images have a first field-of-view (FOV) and a first resolution;

generating a first high-dynamic range (HDR) image from the one or more first images;

receiving, from a second image sensor, one or more second images, wherein the one or more second images have a second FOV and a second resolution;

generating a second high-dynamic range (HDR) image from the one or more second images, including performing tone alignment on the one or more second images based on the first HDR image; and

generating a depth map using the first HDR image and the second HDR image.

16. The method of claim 15 , further comprising:

synchronizing the second FOV of the one or more second images to the first FOV of the one or more first images; and

synchronizing the second resolution of the one or more second images to the first resolution of the one or more first images, and

wherein generating the second HDR image from the one or more second images occurs after synchronizing the second FOV and synchronizing the second resolution.

17. The method of claim 16 , wherein synchronizing the second FOV of the one or more second images to the first FOV of the one or more first images comprises:

causing the second image sensor to crop the second FOV to match the first FOV.

18. The method of claim 17 , wherein synchronizing the second FOV of the one or more second images to the first FOV of the one or more first images further comprises:

spatially aligning the second FOV to match the first FOV.

19. The method of claim 16 , wherein synchronizing the second resolution of the one or more second images to the first resolution of the one or more first images comprises:

causing the second image sensor to perform remosaicing on the one or more second images having the synchronized second FOV.

20. The method of claim 16 , wherein synchronizing the second resolution of the one or more second images to the first resolution of the one or more first images comprises:

causing the second image sensor to perform a binning operation on the one or more second images having the synchronized second FOV.

21. The method of claim 15 , wherein generating the first HDR image from the one or more first images comprises generating the first HDR image using a plurality of the one or more first images using a multi-frame HDR technique, and

wherein generating the second HDR image from the one or more second images comprises generating the second HDR image using a plurality of the one or more second images using the multi-frame HDR technique.

22. The method of claim 15 , wherein generating the first HDR image from the one or more first images comprises generating the first HDR image using a plurality of the one or more first images using a first HDR technique, and

wherein generating the second HDR image from the one or more second images comprises generating the second HDR image using a second HDR technique based on a tone range produced by the first HDR technique.

23. The method of claim 22 , wherein the first HDR technique is a multi-frame HDR technique and wherein the second HDR technique is one of a three-exposure HDR technique or an adaptive dynamic range coding (ADRC) technique.

24. The method of claim 22 , wherein the first HDR technique and the second HDR technique are different techniques.

25. The method of claim 24 , wherein the second HDR technique is an adaptive dynamic range coding (ADRC) technique.

26. The method of claim 15 , further comprising:

applying an image effect to the first HDR image using the depth map.

27. The method of claim 26 , wherein the image effect is a bokeh effect.

28. An apparatus configured to perform camera processing, the apparatus comprising:

means for receiving, from a first image sensor, one or more first images, wherein the one or more first images have a first field-of-view (FOV) and a first resolution;

means for generating a first high-dynamic range (HDR) image from the one or more first images;

means for receiving, from a second image sensor, one or more second images, wherein the one or more second images have a second FOV and a second resolution;

means for generating a second high-dynamic range (HDR) image from the one or more second images, including performing tone alignment on the one or more second images based on the first HDR image; and

means for generating a depth map using the first HDR image and the second HDR image.

29. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to:

receive, from a first image sensor, one or more first images, wherein the one or more first images have a first field-of-view (FOV) and a first resolution;

generate a first high-dynamic range (HDR) image from the one or more first images;

receive, from a second image sensor, one or more second images, wherein the one or more second images have a second FOV and a second resolution;

generate a second high-dynamic range (HDR) image from the one or more second images, including performing tone alignment on the one or more second images based on the first HDR image; and

generate a depth map using the first HDR image and the second HDR image.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 1, 2021
From: LIU, TAI-HSIN; LIU, WEI-CHIH; FENG, WEN-CHUN
To: QUALCOMM INCORPORATED
Reel/Frame 055443/0287 →
Cited By (3)
US 12,389,133 US 12,526,535 US 12,641,211