Image sensor combining high dynamic range techniques
Various technologies described herein pertain to combining high dynamic range techniques to enable rendering higher dynamic range scenes with an image sensor. The image sensor can implement a combination of spatial exposure multiplexing and temporal exposure multiplexing, for example. By way of another example, the image sensor can implement a combination of spatial exposure multiplexing and dual gain operation. Pursuant to another example, the image sensor can implement a combination of temporal exposure multiplexing and dual gain operation. In accordance with yet another example, the image sensor can implement a combination of spatial exposure multiplexing, temporal exposure multiplexing, and dual gain operation. The image sensor can be formed on a single wafer or the image sensor can be a 3D-IC image sensor that includes at least two vertically integrated layers.
1. A three-dimensional integrated circuit (3D-IC) image sensor, comprising:
a sensor layer, comprising:
a pixel array that comprises pixels; and
an image signal processing layer, the image signal processing layer being separate from the sensor layer, the image signal processing layer and the sensor layer being vertically integrated in the 3D-IC image sensor, the image signal processing layer comprising:
a timing controller configured to control exposure times of the pixels in the pixel array, the timing controller configured to:
control a first subset of the pixels in the pixel array to have a first exposure time during a first time period, wherein a first frame of an input image stream is captured during the first time period;
control a second subset of the pixels in the pixel array to have a second exposure time during the first time period;
control the first subset of the pixels in the pixel array to have a third exposure time during a second time period, wherein a second frame of the input image stream is captured during the second time period; and
control the second subset of the pixels in the pixel array to have a fourth exposure time during the second time period;
wherein the first exposure time, the second exposure time, the third exposure time, and the fourth exposure time differ from each other;
a readout circuit configured to read out signals from the pixels in the pixel array for the first frame and the second frame of the input image stream; and
at least one analog-to-digital converter configured to convert the signals to pixel values for the first frame and the second frame of the input image stream;
wherein the first subset of the pixels in the pixel array comprises first non-rectangular units of pixels;
wherein the second subset of the pixels in the pixel array comprises second non-rectangular units of pixels; and
wherein each of the first non-rectangular units of pixels and each of the second non-rectangular units of pixels are L-shaped and comprise three adjacent pixels in a row of the pixel array and a fourth pixel in an adjacent row of the pixel array.
2. The 3D-IC image sensor of claim 1 , wherein:
the 3D-IC image sensor is configured to output the pixel values for the first frame and the second frame of the input image stream to an image signal processor; and
an output frame is generated by the image signal processor based on the pixel values for the first frame and the second frame of the input image stream.
3. The 3D-IC image sensor of claim 1 , wherein the pixel array is a global shutter pixel array.
4. The 3D-IC image sensor of claim 1 , wherein:
the readout circuit is further configured to:
amplify the signals read out from the pixels in the pixel array by a first analog gain to output first amplified signals for the first frame and the second frame of the input image stream; and
amplify the signals read out from the pixels in the pixel array by a second analog gain to output second amplified signals for the first frame and the second frame of the input image stream, wherein the first analog gain differs from the second analog gain; and
the at least one analog-to-digital converter is further configured to:
convert the first amplified signals to first amplified pixel values for the first frame and the second frame of the input image stream; and
convert the second amplified signals to second amplified pixel values for the first frame and the second frame of the input image stream.
5. The 3D-IC image sensor of claim 4 , wherein:
the 3D-IC image sensor is configured to output:
the first amplified pixel values for the first frame and the second frame of the input image stream; and
the second amplified pixel values for the first frame and the second frame of the input image stream; and
an output frame is generated by an image signal processor based on:
the first amplified pixel values for the first frame and the second frame of the input image stream; and
the second amplified pixel values for the first frame and the second frame of the input image stream.
6. The 3D-IC image sensor of claim 4 , wherein the at least one analog-to-digital converter comprises:
column-parallel analog-to-digital converters configured to convert the first amplified signals to the first amplified pixel values for the first frame and the second frame of the input image stream; and
a pipeline analog-to-digital converter configured to convert the second amplified signals to the second amplified pixel values for the first frame and the second frame of the input image stream.
7. The 3D-IC image sensor of claim 1 , the readout circuit comprises:
pixel readouts corresponding to the pixels in the pixel array, wherein the pixel readouts are vertically aligned with and bonded to the corresponding pixels; and
column readouts configured to read out the signals from the pixel readouts on a row-by-row basis.
8. The 3D-IC image sensor of claim 1 , wherein:
the readout circuit comprises buffers configured to amplify the signals read out from the pixels in the pixel array to output amplified signals, each column of the pixel array having a corresponding buffer; and
the at least one analog-to-digital converter comprises one of:
column-parallel analog-to-digital converters configured to convert the amplified signals to amplified pixel values for each column of the pixel array in parallel, each column of the pixel array having a corresponding column-parallel analog-to-digital converter; or
a pipeline analog-to-digital converter configured to convert the amplified signals to the amplified pixel values.
9. The 3D-IC image sensor of claim 1 , wherein:
the readout circuit comprises:
high gain buffers configured to amplify the signals read out from the pixels in the pixel array by a first analog gain to output first amplified signals, each column of the pixel array having a corresponding high gain buffer; and
low gain buffers configured to amplify the signals read out from the pixels in the pixel array by a second analog gain to output second amplified signals, each column in the pixel array having a corresponding low gain buffer; and
the at least one analog-to-digital converter comprises:
one or more analog-to-digital converters configured to convert the first amplified signals to first amplified pixel values; and
one or more analog-to-digital converters configured to convert the second amplified signals to second amplified pixel values.
10. The 3D-IC image sensor of claim 1 , wherein:
the sensor layer is a P-type metal-oxide-semiconductor (PMOS) layer; and
the image signal processing layer is a complementary metal-oxide-semiconductor (CMOS) layer.
11. A three-dimensional integrated circuit (3D-IC) image sensor, comprising:
a sensor layer, comprising:
a pixel array that comprises pixels;
an image signal processing layer, the image signal processing layer being separate from the sensor layer, the image signal processing layer and the sensor layer being vertically integrated in the 3D-IC image sensor, the image signal processing layer comprising:
a timing controller configured to control exposure times of the pixels in the pixel array, the timing controller configured to control the exposure times of the pixels at least one of temporally or spatially such that an input image stream has two or more differing exposure times;
a readout circuit configured to:
read out signals from the pixels in the pixel array for the input image stream having the two or more differing exposure times;
amplify the signals read out from the pixels in the pixel array by a first analog gain to output first amplified signals for the input image stream having the two or more differing exposure times; and
amplify the signals read out from the pixels in the pixel array by a second analog gain to output second amplified signals for the input image stream having the two or more differing exposure times, wherein the first analog gain differs from the second analog gain; and
at least one analog-to-digital converter configured to:
convert the first amplified signals to first amplified pixel values for the input image stream having the two or more differing exposure times; and
convert the second amplified signals to second amplified pixel values for the input image stream having the two or more differing exposure times;
wherein the timing controlling being configured to control the exposure times of the pixels both temporally and spatially comprises the timing controller being configured to:
control a first subset of the pixels in the pixel array to have a first exposure time during a first time period, wherein a first frame of the input image stream is captured during the first time period;
control a second subset of the pixels in the pixel array to have a second exposure time during the first time period;
control the first subset of the pixels in the pixel array to have a third exposure time during a second time period, wherein a second frame of the input image stream is captured during the second time period; and
control the second subset of the pixels in the pixel array to have a fourth exposure time during the second time period;
wherein the first exposure time, the second exposure time, the third exposure time, and the fourth exposure time differ from each other;
wherein the first subset of the pixels in the pixel array comprises first non-rectangular units of pixels;
wherein the second subset of the pixels in the pixel array comprises second non-rectangular units of pixels; and
wherein each of the first non-rectangular units of pixels and each of the second non-rectangular units of pixels are L-shaped and comprise three adjacent pixels in a row of the pixel array and a fourth pixel in an adjacent row of the pixel array.
12. The 3D-IC image sensor of claim 11 , wherein:
an output frame is generated by an image signal processor based on:
the first amplified pixel values for the input image stream having the two or more differing exposure times; and
the second amplified pixel values for the input image stream having the two or more differing exposure times.
13. The 3D-IC image sensor of claim 11 , wherein the pixel array is a global shutter pixel array.
14. An image sensor, comprising:
a pixel array that comprises pixels;
a timing controller configured to control exposure times of the pixels in the pixel array, the timing controller configured to:
control a first subset of the pixels in the pixel array to have a first exposure time during a first time period, wherein a first frame of an input image stream is captured during the first time period;
control a second subset of the pixels in the pixel array to have a second exposure time during the first time period;
control the first subset of the pixels in the pixel array to have a third exposure time during a second time period, wherein a second frame of the input image stream is captured during the second time period; and
control the second subset of the pixels in the pixel array to have a fourth exposure time during the second time period;
wherein the first exposure time, the second exposure time, the third exposure time, and the fourth exposure time differ from each other;
a readout circuit configured to read out signals from the pixels in the pixel array for the first frame and the second frame of the input image stream; and
at least one analog-to-digital converter configured to convert the signals to pixel values for the first frame and the second frame of the input image stream;
wherein the pixel array, the timing controller, the readout circuit, and the at least one analog-to-digital converter are formed on a single wafer;
wherein the first subset of the pixels in the pixel array comprises first non-rectangular units of pixels;
wherein the second subset of the pixels in the pixel array comprises second non-rectangular units of pixels; and
wherein each of the first non-rectangular units of pixels and each of the second non-rectangular units of pixels are L-shaped and comprise three adjacent pixels in a row of the pixel array and a fourth pixel in an adjacent row of the pixel array.
15. The image sensor of claim 14 , wherein:
the readout circuit is further configured to:
amplify the signals read out from the pixels in the pixel array by a first analog gain to output first amplified signals for the first frame and the second frame of the input image stream; and
amplify the signals read out from the pixels in the pixel array by a second analog gain to output second amplified signals for the first frame and the second frame of the input image stream, wherein the first analog gain differs from the second analog gain; and
the at least one analog-to-digital converter is further configured to:
convert the first amplified signals to first amplified pixel values for the first frame and the second frame of the input image stream; and
convert the second amplified signals to second amplified pixel values for the first frame and the second frame of the input image stream.