Image sensor combining high dynamic range techniques
Various technologies described herein pertain to combining high dynamic range techniques to enable rendering higher dynamic range scenes with an image sensor. The image sensor can implement a combination of spatial exposure multiplexing and temporal exposure multiplexing, for example. By way of another example, the image sensor can implement a combination of spatial exposure multiplexing and dual gain operation. Pursuant to another example, the image sensor can implement a combination of temporal exposure multiplexing and dual gain operation. In accordance with yet another example, the image sensor can implement a combination of spatial exposure multiplexing, temporal exposure multiplexing, and dual gain operation.
1. An image sensor, comprising:
a pixel array that comprises pixels;
a timing controller configured to control exposure times of the pixels in the pixel array, the timing controller configured to:
control a first subset of the pixels in the pixel array to have a first exposure time during a first time period, wherein a first frame of an input image stream is captured during the first time period;
control a second subset of the pixels in the pixel array to have a second exposure time during the first time period;
control the first subset of the pixels in the pixel array to have a third exposure time during a second time period, wherein a second frame of the input image stream is captured during the second time period; and
control the second subset of the pixels in the pixel array to have a fourth exposure time during the second time period;
wherein the first exposure time, the second exposure time, the third exposure time, and the fourth exposure time differ from each other;
a readout circuit configured to read out signals from the pixels in the pixel array for the first frame and the second frame of the input image stream; and
an analog-to-digital converter configured to convert the signals to pixel values for the first frame and the second frame of the input image stream;
wherein the first subset of the pixels in the pixel array comprise first non-rectangular units of pixels;
wherein the second subset of the pixels in the pixel array comprise second non-rectangular units of pixels;
wherein the first non-rectangular units of pixels oppose the second non-rectangular units of pixels to form a zigzag pattern in the pixel array; and
wherein each of the first non-rectangular units of pixels and each of the second non-rectangular units of pixels are L-shaped and comprise three adjacent pixels in a row of the pixel array and a fourth pixel in an adjacent row of the pixel array.
2. The image sensor of claim 1 , further comprising:
an image signal processor configured to generate an output frame based on the pixel values for the first frame and the second frame of the input image stream.
3. The image sensor of claim 1 , wherein:
the image sensor is configured to output the pixel values for the first frame and the second frame of the input image stream to an image signal processor; and
an output frame is generated by an image signal processor based on the pixel values for the first frame and the second frame of the input image stream.
4. The image sensor of claim 1 , wherein:
the readout circuit is further configured to:
amplify the signals read out from the pixels in the pixel array by a first analog gain to output first amplified signals for the first frame and the second frame of the input image stream; and
amplify the signals read out from the pixels in the pixel array by a second analog gain to output second amplified signals for the first frame and the second frame of the input image stream, wherein the first analog gain differs from the second analog gain; and
the analog-to-digital converter is further configured to:
convert the first amplified signals to first amplified pixel values for the first frame and the second frame of the input image stream; and
convert the second amplified signals to second amplified pixel values for the first frame and the second frame of the input image stream.
5. The image sensor of claim 4 , further comprising:
an image signal processor configured to generate an output frame based on:
the first amplified pixel values for the first frame and the second frame of the input image stream; and
the second amplified pixel values for the first frame and the second frame of the input image stream.
6. The image sensor of claim 4 , wherein:
the image sensor is configured to output:
the first amplified pixel values for the first frame and the second frame of the input image stream; and
the second amplified pixel values for the first frame and the second frame of the input image stream; and
an output frame is generated by an image signal processor based on:
the first amplified pixel values for the first frame and the second frame of the input image stream; and
the second amplified pixel values for the first frame and the second frame of the input image stream.
7. An image sensor, comprising:
a pixel array that comprises pixels, wherein the pixel array comprises a first subset of the pixels and a second subset of the pixels, wherein the first subset of the pixels in the pixel array comprise first non-rectangular units of pixels, wherein the second subset of the pixels in the pixel array comprise second non-rectangular units of pixels, wherein the first non-rectangular units of pixels oppose the second non-rectangular units of pixels to form a zigzag pattern in the pixel array, and wherein each of the first non-rectangular units of pixels and each of the second non-rectangular units of pixels are L-shaped and comprise three adjacent pixels in a row of the pixel array and a fourth pixel in an adjacent row of the pixel array;
a timing controller configured to control exposure times of the pixels in the pixel array, the timing controller configured to control the exposure times of the pixels at least one of temporally or spatially such that an input image stream has two or more differing exposure times;
a readout circuit configured to:
read out signals from the pixels in the pixel array for the input image stream having the two or more differing exposure times;
amplify the signals read out from the pixels in the pixel array by a first analog gain to output first amplified signals for the input image stream having the two or more differing exposure times; and
amplify the signals read out from the pixels in the pixel array by a second analog gain to output second amplified signals for the input image stream having the two or more differing exposure times, wherein the first analog gain differs from the second analog gain; and
an analog-to-digital converter configured to:
convert the first amplified signals to first amplified pixel values for the input image stream having the two or more differing exposure times; and
convert the second amplified signals to second amplified pixel values for the input image stream having the two or more differing exposure times.
8. The image sensor of claim 7 , the timing controller being configured to control the exposure times of the pixels temporally comprises the timing controller being configured to:
control the pixels in the pixel array to have a first exposure time during a first time period, wherein a first frame of the input image stream is captured during the first time period; and
control the pixels in the pixel array to have a second exposure time during a second time period, wherein a second frame of the input image stream is captured during the second time period;
wherein the first exposure time and the second exposure time differ from each other.
9. The image sensor of claim 7 , the timing controller being configured to control the exposure times of the pixels spatially comprises the timing controller being configured to:
control the first subset of the pixels in the pixel array to have a first exposure time during a time period, wherein a frame of the input image stream is captured during the time period; and
control the second subset of the pixels in the pixel array to have a second exposure time during the time period; and
wherein the first exposure time and the second exposure time differ from each other.
10. The image sensor of claim 7 , the timing controller being configured to control the exposure times of the pixels both temporally and spatially comprises the timing controller being configured to:
control the first subset of the pixels in the pixel array to have a first exposure time during a first time period, wherein a first frame of the input image stream is captured during the first time period;
control the second subset of the pixels in the pixel array to have a second exposure time during the first time period;
control the first subset of the pixels in the pixel array to have a third exposure time during a second time period, wherein a second frame of the input image stream is captured during the second time period; and
control the second subset of the pixels in the pixel array to have a fourth exposure time during the second time period;
wherein the first exposure time, the second exposure time, the third exposure time, and the fourth exposure time differ from each other.
11. The image sensor of claim 7 , further comprising:
an image signal processor configured to generate an output frame based on:
the first amplified pixel values for the input image stream having the two or more differing exposure times; and
the second amplified pixel values for the input image stream having the two or more differing exposure times.
12. The image sensor of claim 7 , wherein:
an output frame is generated by an image signal processor based on:
the first amplified pixel values for the input image stream having the two or more differing exposure times; and
the second amplified pixel values for the input image stream having the two or more differing exposure times.
13. A method of increasing dynamic range of an image sensor, the image sensor comprising a pixel array that comprises pixels, the method comprising:
controlling a first subset of the pixels in the pixel array to have a first exposure time during a first time period, wherein a first frame of an input image stream is captured during the first time period;
controlling a second subset of the pixels in the pixel array to have a second exposure time during the first time period;
reading out signals from the pixels in the pixel array for the first frame of the input image stream;
converting the signals for the first frame to digital pixel values for the first frame of the input image stream;
controlling the first subset of the pixels in the pixel array to have a third exposure time during a second time period, wherein a second frame of the input image stream is captured during the second time period;
controlling the second subset of the pixels in the pixel array to have a fourth exposure time during the second time period, wherein the first exposure time, the second exposure time, the third exposure time, and the fourth exposure time differ from each other;
reading out signals from the pixels in the pixel array for the second frame of the input image stream; and
converting the signals for the second frame to digital pixel values for the second frame of the input image stream;
wherein the first subset of the pixels in the pixel array comprise first non-rectangular units of pixels;
wherein the second subset of the pixels in the pixel array comprise second non-rectangular units of pixels;
wherein the first non-rectangular units of pixels oppose the second non-rectangular units of pixels to form a zigzag pattern in the pixel array; and
wherein each of the first non-rectangular units of pixels and each of the second non-rectangular units of pixels are L-shaped and comprise three adjacent pixels in a row of the pixel array and a fourth pixel in an adjacent row of the pixel array.
14. The method of claim 13 , further comprising:
generating an output frame based on the digital pixel values for the first frame of the input image stream and the digital pixel values for the second frame of the input image stream.