Timestamp calibration of the 3D camera with epipolar line laser point scanning
Using the same image sensor to capture a two-dimensional (2D) image and three-dimensional (3D) depth measurements for a 3D object. A laser point-scans the surface of the object with light spots, which are detected by a pixel array in the image sensor to generate the 3D depth profile of the object using triangulation. Each row of pixels in the pixel array forms an epipolar line of the corresponding laser scan line. Timestamping provides a correspondence between the pixel location of a captured light spot and the respective scan angle of the laser to remove any ambiguity in triangulation. An Analog-to-Digital Converter (ADC) in the image sensor operates as a Time-to-Digital (TDC) converter to generate timestamps. A timestamp calibration circuit is provided on-board to record the propagation delay of each column of pixels in the pixel array and to provide necessary corrections to the timestamp values generated during 3D depth measurements.
1. An imaging unit comprising:
a light source operative to perform a one-dimensional (1D) point scan of a three-dimensional (3D) object along a scanning line, wherein the point scan projects a sequence of light spots on a surface of the 3D object; and
an image sensor unit that includes:
a plurality of pixels arranged in a two-dimensional (2D) pixel array forming an image plane, wherein a row of pixels in the 2D pixel array forms at least a portion of an epipolar line of the scanning line, wherein each pixel in the row of pixels is associated with a respective column in the 2D pixel array, and wherein each pixel in the row of pixels is operative to detect a corresponding light spot in the sequence of light spots,
a plurality of Analog-to-Digital Converter (ADC) units, wherein each ADC unit is associated with a respective pixel in the row of pixels and is operative to generate a pixel-specific timestamp value for the respective pixel in response to a pixel-specific detection of the corresponding light spot by the respective pixel, and
a processing unit coupled to the plurality of ADC units, wherein the processing unit is operative to perform the following:
for a column in the 2D pixel array associated with the respective pixel in the row of pixels, apply a column-specific correction value to the pixel-specific timestamp value to obtain a corrected timestamp value, wherein the column-specific correction value represents a column-specific propagation delay between the pixel-specific detection and when a pixel-specific output of the respective pixel reaches a pre-defined threshold, and
determine a distance to the corresponding light spot on the surface of the 3D object based at least on the corrected timestamp value and on a scan angle used by the light source for projecting the corresponding light spot.
2. The imaging unit of claim 1 , wherein the 2D pixel array is a Complementary Metal Oxide Semiconductor (CMOS) array, and wherein each pixel in the row of pixels is one of the following:
a Four Transistor (4T) pixel;
a Three Transistor (3T) pixel;
a Two Transistor (2T) pixel;
a One Transistor (1T) pixel;
a shared-transistor pixel;
a 1×2 2-shared pixel; and
a 1×4 4-shared pixel.
3. The imaging unit of claim 1 , wherein the light source is one of the following:
a laser light source;
a visible laser light source;
a near infrared laser light source;
a point light source;
a monochromatic illumination source; and
an X-Y addressable light source.
4. The imaging unit of claim 1 , wherein the image sensor unit further includes:
a calibration unit operatively coupled to the plurality of ADC units and the processing unit to simulate the column-specific propagation delay in hardware to determine the column-specific correction value prior to generation of the pixel-specific timestamp value.
5. The imaging unit of claim 1 , wherein the image sensor unit further includes:
a calibration unit coupled to the plurality of ADC units and the processing unit, wherein the calibration unit includes a column-specific calibration pixel for the column associated with the respective pixel, and wherein the calibration pixel is in addition to the respective pixel in the 2D pixel array,
and wherein the following applies:
the calibration unit is operative to activate the calibration pixel and clock an output of the calibration pixel in the same manner as an output from the respective pixel that is associated with the calibration pixel is clocked;
one of the plurality of ADC units is operative to record a first timing instant when the calibration pixel is activated and to record a second timing instant when the output of the calibration pixel reaches the pre-defined threshold; and
the processing unit is operative to subtract the first timing instant from the second timing instant to obtain the column-specific correction value.
6. The imaging unit of claim 5 , wherein the calibration unit is operative to perform one of the following:
clock the output of the calibration pixel using a first clock frequency that is equal to a scan frequency used by the light source in performing the 1D point scan; and
clock the output of the calibration pixel using a second clock frequency that is different from the scan frequency.
7. The imaging unit of claim 1 , wherein the image sensor unit further includes:
a calibration unit coupled to the plurality of ADC units and the processing unit, wherein the calibration unit includes a column-specific calibration pixel for the column associated with the respective pixel, and wherein the calibration pixel is in addition to the respective pixel in the 2D pixel array,
and wherein the following applies:
the calibration unit is operative to activate the calibration pixel and clock an output of the calibration pixel in the same manner as an output from the respective pixel that is associated with the calibration pixel is clocked;
one of the plurality of ADC units is operative to commence a counting operation substantially simultaneously with activation of the calibration pixel and terminate the counting operation when the output of the calibration pixel reaches the pre-defined threshold; and
the processing unit is operative to select a count value generated upon termination of the counting operation as the column-specific correction value.
8. A system comprising:
a light source operative to perform a one-dimensional (1D) point scan of a three-dimensional (3D) object along a scanning line, wherein the point scan projects a sequence of light spots on a surface of the 3D object;
a plurality of pixels arranged in a two-dimensional (2D) pixel array forming an image plane, wherein a row of pixels in the 2D pixel array forms at least a portion of an epipolar line of the scanning line, wherein each pixel in the row of pixels is associated with a respective column in the 2D pixel array, and wherein each pixel in the row of pixels is operative to detect a corresponding light spot in the sequence of light spots;
a plurality of Analog-to-Digital Converter (ADC) units, wherein each ADC unit is associated with a respective pixel in the row of pixels and is operative to generate a pixel-specific timestamp value for the respective pixel in response to a pixel-specific detection of the corresponding light spot by the respective pixel;
a memory for storing program instructions; and
a processor coupled to the memory and to the plurality of ADC units, wherein the processor is configured to execute the program instructions, whereby the processor is operative to perform the following:
for a column in the 2D pixel array associated with the respective pixel in the row of pixels, apply a column-specific correction value to the pixel-specific timestamp value to obtain a corrected timestamp value, wherein the column-specific correction value represents a column-specific propagation delay between the pixel-specific detection and when a pixel-specific output of the respective pixel reaches a pre-defined threshold, and
determine a distance to the corresponding light spot on the surface of the 3D object based at least on the corrected timestamp value and on a scan angle used by the light source for projecting the corresponding light spot.
9. The system of claim 8 , wherein the 2D pixel array is a Complementary Metal Oxide Semiconductor (CMOS) array, and wherein each pixel in the row of pixels is one of the following:
a Four Transistor (4T) pixel;
a Three Transistor (3T) pixel;
a Two Transistor (2T) pixel;
a One Transistor (1T) pixel;
a shared-transistor pixel;
a 1×2 2-shared pixel; and
a 1×4 4-shared pixel.
10. The system of claim 8 , wherein the light source is one of the following:
a laser light source;
a visible light source;
a point light source;
a monochromatic illumination source; and
an X-Y addressable light source.
11. The system of claim 8 , further comprising:
a calibration unit operatively coupled to the plurality of ADC units and the processor to simulate the column-specific propagation delay in hardware to determine the column-specific correction value.
12. The system of claim 8 , further comprising:
a calibration unit coupled to the plurality of ADC units and the processor, wherein the calibration unit includes a column-specific calibration pixel for the column associated with the respective pixel, and wherein the calibration pixel is in addition to the respective pixel in the 2D pixel array,
and wherein, upon execution of the program instructions, the processor is operative to further perform the following:
configure the calibration unit to activate the calibration pixel and clock an output of the calibration pixel in the same manner as an output from the respective pixel that is associated with the calibration pixel is clocked;
configure one of the plurality of ADC units to record a first timing instant when the calibration pixel is activated and to record a second timing instant when the output of the calibration pixel reaches the pre-defined threshold; and
subtract the first timing instant from the second timing instant to obtain the column-specific correction value.
13. The system of claim 12 , wherein the ADC unit recording the second timing instant is operative to perform one of the following:
commence generation of a corresponding timestamp upon occurrence of the second timing instant; and
conclude generation of the corresponding timestamp upon occurrence of the second timing instant.
14. The system of claim 12 , wherein the calibration unit is operative to perform one of the following:
receive a first clock signal to clock the output of the calibration pixel, wherein the first clock signal has a first clock frequency that is equal to a scan frequency used by the light source in performing the 1D point scan; and
receive a second clock signal to clock the output of the calibration pixel, wherein the second clock signal has a second clock frequency that is different from the scan frequency.
15. The system of claim 8 , further comprising:
a calibration unit coupled to the plurality of ADC units and the processor, wherein the calibration unit includes a column-specific calibration pixel for the column associated with the respective pixel, and wherein the calibration pixel is in addition to the respective pixel in the 2D pixel array,
and wherein, upon execution of the program instructions, the processor is operative to further perform the following:
configure the calibration unit to activate the calibration pixel and clock an output of the calibration pixel in the same manner as an output from the respective pixel that is associated with the calibration pixel is clocked;
configure one of the plurality of ADC units to commence a counting operation prior to or substantially simultaneously with activation of the calibration pixel and terminate the counting operation when the output of the calibration pixel reaches the pre-defined threshold; and
select a count value generated upon termination of the counting operation as the column-specific correction value.
16. The system of claim 8 , wherein the system is one of the following:
a mobile device;
a smartphone;
a tablet computer;
a computing unit;
a User Equipment (UE);
an automobile;
a virtual reality equipment; and
a robot.