Granted Patent B2

US 12,244,943 · App. 18/195,208 · Granted Mar 4, 2025

Inventors: Yang-Ting Chou (Zhubei, TW); Zhaojian Li (Fremont, CA)

Assignee: Cista System Corp.

H04N25/57G06T5/10G06T5/50G06T5/70G06T5/92H04N25/11H04N25/62G06T2207/10024G06T2207/20064G06T2207/20208G06T2207/20221

Abstract

Claims (74)

1. A method for High Dynamic Range (HDR) using M×N-cell image sensor, comprising:

capturing, using the M×N-cell image sensor, a plurality of frames of a scene using a short-exposure setting, wherein:

the M×N-cell image sensor comprises a plurality of pixel groups, each pixel group having M×N pixels configured with color filters of a same color such that the M×N pixels in each pixel group capture the same color,

M and N are integers greater than one, and

each pixel is represented by a first number of bits;

merging the plurality of frames into an HDR image, wherein the merging comprises:

identifying a pixel group from each of the plurality of frames; and

merging the M×N pixels in each of the identified pixel groups into one super-pixel in the HDR image, wherein each super-pixel in the HDR image is represented by a second number of bits, the second number being greater than the first number; and

performing tone mapping on the HDR image to reduce dynamic range of the super-pixels of the HDR image into a low dynamic range (LDR) image, wherein each pixel in the LDR image is represented with the first number of bits.

2. The computer-implemented method of claim 1 , wherein the M×N pixels in each pixel group comprise M rows and N columns of pixels.

3. The computer-implemented method of claim 1 , wherein M equals N, such that each pixel group is a square shape comprising at least four pixels.

4. The computer-implemented method of claim 1 , wherein the method further comprises determining a quantity of the plurality of frames to capture by:

determining a target dynamic range to achieve in the HDR image, wherein the target dynamic range corresponds to the second number of bits;

determining a pixel-bit difference between the first number of bits and the second number of bits;

determining a number of frames to capture to achieve the target dynamic range based on the pixel-bit difference; and

reducing the number of frames to capture based on a quantity of pixels in each pixel group in the M×N-cell image sensor.

5. The computer-implemented method of claim 4 , wherein the determining the number of frames to capture to achieve the target dynamic range based on the pixel-bit difference comprises:

determining a first number of frames to capture as a power of the pixel-bit difference for reducing shot noises in captured frames;

determining a second number of frames to capture for reducing read noise in the captured frames introduced by the M×N-cell image sensor every time a frame is captured; and

determining the number of frames to capture as a sum of the first number and the second number.

6. The computer-implemented method of claim 1 , wherein the identifying a pixel group from each of the plurality of frames comprises

selecting a pixel group from a same location in each of the plurality of frames.

7. The computer-implemented method of claim 6 , wherein the merging the M×N pixels in each of the identified pixel groups into one super-pixel in the HDR image comprises:

decomposing each of the plurality of frames into a set of frequency bands using a wavelet transform algorithm; and

merging pixel groups from each same frequency band using an inverse wavelet transform algorithm.

8. The computer-implemented method of claim 7 , further comprising:

performing denoising on each wavelet band to obtain the super-pixels.

9. The computer-implemented method of claim 1 , wherein the identifying a pixel group from each of the plurality of frames comprises:

comparing a pixel group from a first frame with all pixel groups in other frames to identify similar pixel groups.

10. The computer-implemented method of claim 1 , wherein the tone mapping comprises:

applying different tone mapping curves to different areas of the HDR image.

11. The computer-implemented method of claim 10 , wherein the tone mapping further comprises:

constructing a fusion tensor comprising a plurality of weights corresponding to the different tone mapping curves;

mapping super-pixels in the HDR image to the fusion tensor to obtain corresponding fusion weights; and

performing image fusion on the HDR image based on the plurality of fusion weights.

12. An image sensor apparatus, comprising:

a plurality of pixel groups, each pixel group having M×N pixels configured with color filters of a same color such that the M×N pixels in each pixel group capture a same color, wherein M and N are integers greater than one, and each pixel is represented by a first number of bits; and

one or more processors configured to:

capture, through the plurality of pixel groups, a plurality of frames of a scene using a short-exposure setting;

merge the plurality of frames into an HDR image, wherein the merging comprises:

identifying a pixel group from each of the plurality of frames; and

merging the M×N pixels in each of the identified pixel groups into one super-pixel in the HDR image, wherein each super-pixel in the HDR image is represented by a second number of bits, the second number being greater than the first number; and

perform tone mapping on the HDR image to reduce a dynamic range of the super-pixels of the HDR image into a low dynamic range (LDR) image, wherein each pixel in the LDR image is represented with the first number of bits.

13. The image sensor apparatus of claim 12 , wherein M equals N, such that each pixel group is a square shape comprising at least four pixels.

14. The image sensor apparatus of claim 12 , wherein the one or more processors are further configured to:

determine a target dynamic range to achieve in the HDR image, wherein the target dynamic range corresponds to the second number of bits;

determine a pixel-bit difference between the first number of bits and the second number of bits;

determine a number of frames to capture to achieve the target dynamic range based on the pixel-bit difference; and

reduce the number of frames to capture based on a quantity of pixels in each pixel group in the M×N image sensor apparatus.

15. The image sensor apparatus of claim 14 , wherein to determine the number of frames to capture to achieve the target dynamic range, the one or more processors are further configured to:

determine a first number of frames to capture as a power of the pixel-bit difference for reducing shot noises in captured frames;

determine a second number of frames to capture for reducing read noise in the captured frames introduced by the M×N image sensor apparatus every time a frame is captured; and

determine the number of frames to capture as a sum of the first number and the second number.

16. The image sensor apparatus of claim 12 , wherein to identify a pixel group from each of the plurality of frames, the one or more processors are further configured to:

select a pixel group from a same location in each of the plurality of frames.

17. The image sensor apparatus of claim 12 , wherein to identify a pixel group from each of the plurality of frames, the one or more processors are further configured to:

compare a pixel group in a first frame with all pixel groups in other frames to identify similar pixel groups.

18. The image sensor apparatus of claim 12 , wherein tone mapping comprises:

applying different tone mapping curves to different areas of the HDR image.

19. A non-transitory computer-readable storage medium configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising:

capturing, using a M×N-cell image sensor, a plurality of frames of a scene using a short-exposure setting, wherein:

the M×N-cell image sensor comprises a plurality of pixel groups, each pixel group having M×N pixels configured with color filters of a same color such that the M×N pixels in each pixel group capture the same color,

M and N are integers greater than one, and

each pixel is represented by a first number of bits;

merging the plurality of frames into an HDR image, wherein the merging comprises:

identifying a pixel group from each of the plurality of frames; and

merging the M×N pixels in each of the identified pixel groups into one super-pixel in the HDR image, wherein each super-pixel in the HDR image is represented by a second number of bits, the second number being greater than the first number; and

performing tone mapping on the HDR image to reduce a dynamic range of the super-pixels of the HDR image into a low dynamic range (LDR) image, wherein each pixel in the LDR image is represented with the first number of bits.

20. The non-transitory computer-readable storage medium of claim 19 , wherein the operations further comprise:

determining a quantity of the plurality of frames to capture by:

determining a target dynamic range to achieve in the HDR image, wherein the target dynamic range corresponds to the second number of bits;

determining a pixel-bit difference between the first number of bits and the second number of bits;

determining a number of frames to capture to achieve the target dynamic range based on the pixel-bit difference; and

reducing the number of frames to capture based on a quantity of pixels in each pixel group in the M×N-cell image sensor.

Assignments (1)

Continuity (1)

References Cited (16)