Light-emitting diode (LED) flicker mitigation (LFM) using spatially multiplexed image sensor

1 . An M×N-cell image sensor, comprising:

a plurality of pixels, each pixel being divided into M×N sub-pixels, and each of the M×N sub-pixels being configured with an exposure time and an analog gain, wherein M and N are positive integers greater than one;

wherein:

within each of the plurality of pixels:

the M×N sub-pixels are configured with a plurality of different exposure times;

a first sub-pixel of the M×N sub-pixels is configured with a first exposure time and a first analog gain, and a second sub-pixel of the M×N sub-pixels is configured with a second exposure time and a second analog gain, wherein the first exposure time is longer than the second exposure time, and the first analog gain is smaller than the second analog gain; and

the first sub-pixel and the second sub-pixel have same energy, which is computed as a product of exposure time and analog gain.

2 . The M×N-cell image sensor of claim 1 , wherein:

the first sub-pixel is selected from the M×N sub-pixels except for a sub-pixel with a shortest exposure time among the M×N sub-pixels.

3 . The M×N-cell image sensor of claim 1 , wherein:

the first sub-pixel is a sub-pixel with a second shortest exposure time among the M×N sub-pixels, and the second sub-pixel is a sub-pixel with a shortest exposure time among the M×N sub-pixels.

4 . The M×N-cell image sensor of claim 3 , wherein:

the first exposure time of the first sub-pixel is m times greater than the second exposure time of the second sub-pixel, wherein m is an integer greater than one, and

the first analog gain of the first sub-pixel is configured as

1

m

⁢

th

 of the second analog gain of the second sub-pixel.

5 . The M×N-cell image sensor of claim 1 , further comprising a processor configured to:

accumulate pixel values of the first sub-pixel and the second sub-pixel to obtain an intermediate pixel value, thereby detecting different flicker positions through different exposure times.

6 . The M×N-cell image sensor of claim 5 , wherein the processor is further configured to:

perform High Dynamic Range (HDR) fusion of the intermediate pixel value and pixel values of other M×N sub-pixels when flicker is not detected.

7 . The M×N-cell image sensor of claim 5 , wherein the processor is further configured to:

perform LED flicker mitigation (LFM) during HDR fusion of the intermediate pixel value and pixel values of other M×N sub-pixels when a flicker is detected.

8 . The M×N-cell image sensor of claim 1 , wherein the M×N sub-pixels are configured with linearly increasing exposure times.

9 . The M×N-cell image sensor of claim 1 , wherein, except for the first sub-pixel, all other sub-pixels in the M×N sub-pixels are configured with the second analog gain.

10 . An LED flicker mitigation (LFM) method, comprising:

capturing High Dynamic Range (HDR) frames by using an M×N-cell image sensor, M and N being positive integers greater than one, wherein:

the M×N-cell image sensor comprises a plurality of pixels,

each pixel being divided into M×N sub-pixels, and

within each of the plurality of pixels:

the M×N sub-pixels are configured with a plurality of different exposure times; and

a first sub-pixel of the M×N sub-pixels is configured with a first exposure time and a first analog gain, and a second sub-pixel of the M×N sub-pixels is configured with a second exposure time and a second analog gain,

wherein the first exposure time is longer than the second exposure time,

the first analog gain is smaller than the second analog gain, and

the first sub-pixel and the second sub-pixel have same energy, which is computed as a product of exposure time and analog gain;

detecting LED flickering based on accumulating a pixel value of the first sub-pixel and a pixel value of the second sub-pixel in the M×N sub-pixels; and

in response to LED flickering being detected, performing LED flicker mitigation (LFM) during HDR fusion of the HDR frames.

11 . The LFM method of claim 10 , wherein:

except for the first sub-pixel, all other sub-pixels in the M×N sub-pixels are configured with the second analog gain.

12 . The LFM method of claim 10 , wherein:

the first sub-pixel is a sub-pixel that has a second shortest exposure time among the M×N sub-pixels, and the second sub-pixel is a sub-pixel with a shortest exposure time among the M×N sub-pixels.

13 . The LFM method of claim 12 , wherein:

the first exposure time of the first sub-pixel is m times greater than the second exposure time of the second sub-pixel, wherein m is an integer greater than one, and

the first analog gain of the first sub-pixel is configured as

1

m

⁢

th

 of the second analog gain of the second sub-pixel,

wherein the pixel value of the first sub-pixel is determined based on the first exposure time and the first analog gain, and

the pixel value of the second sub-pixel is determined based on the second exposure time and the second analog gain.

14 . The LFM method of claim 10 , wherein the performing LFM during fusion of the HDR frames comprises:

performing LFM at pixels that captured the LED flickering, and

performing HDR fusion at pixels that did not capture the LED flickering.

15 . The LFM method of claim 10 , wherein the M×N sub-pixels are configured with linearly increasing exposure times.

16 . A non-transitory computer-readable storage medium configured with instructions executable by one or more processors to cause the one or more processors to perform operations comprising:

capturing High Dynamic Range (HDR) frames by using an M×N-cell image sensor, M and N being positive integers greater than one, wherein:

the M×N-cell image sensor comprises a plurality of pixels,

each pixel being divided into M×N sub-pixels,

within each of the plurality of pixels:

the M×N sub-pixels are configured with a plurality of different exposure times; and

a first sub-pixel of the M×N sub-pixels is configured with a first exposure time and a first analog gain, and a second sub-pixel of the M×N sub-pixels is configured with a second exposure time and a second analog gain, wherein the first exposure time is longer than the second exposure time, the first analog gain is smaller than the second analog gain, and the first sub-pixel and the second sub-pixel have same energy, which is computed as a product of exposure time and analog gain;

detecting LED flickering based at least on accumulating a pixel value of the first sub-pixel and a pixel value of the second sub-pixel in the M×N sub-pixels; and

based on the detected LED flickering, performing LED flicker mitigation (LFM) during HDR fusion of the HDR frames.

17 . The non-transitory computer-readable storage medium of claim 16 , wherein the first sub-pixel is selected from the M×N sub-pixels except for a sub-pixel with a shortest exposure time among the M×N sub-pixels.

18 . The non-transitory computer-readable storage medium of claim 16 , wherein the first sub-pixel is a sub-pixel with a second shortest exposure time among the M×N sub-pixels, and the second sub-pixel is a sub-pixel with a shortest exposure time among the M×N sub-pixels.

19 . The non-transitory computer-readable storage medium of claim 18 , wherein:

the first exposure time of the first sub-pixel is m times greater than the second exposure time one of the second sub-pixel, and

the first analog gain of the first sub-pixel is configured as

1

m

⁢

th

 of the second analog gain of the second sub-pixel.