Single image sensor for capturing mixed structured-light images and regular images
A method and device for improving the accuracy of depth information derived from a structured-light image for a regular image are disclosed. In one example, an additional structured-light image is captured before a first structured-light image or after a regular image. The depth information for the regular image can be derived from the first structured-light image and corrected by incorporating depth information from the additional structured-light image. A model for depth information can be used to predict or interpolate depth information for the regular image. In another example, two regular sub-images may be captured with a structured-light image in between. If substantial frame differences or substantial global motion vector/block motion vectors are detected, the two regular sub-images will not be combined in order to avoid possible motion smear. Instead, one of the two sub-images will be selected and scaled as the output regular image.
1. A method of deriving depth information for a regular image from structured-light images using a camera comprising an image sensor, the method comprising:
projecting a first structured light to a scene in a field of view of the image sensor, wherein the image sensor comprises a pixel array;
applying first reset signals to the pixel array to reset rows of the pixel array;
exposing the rows of the pixel array to the first structured light to cause first analog signals from the rows of the pixel array;
capturing a first structured-light image using the pixel array during a first frame period by converting the first analog signals from the rows of the pixel array into first digital outputs to form the first structured-light image using one or more analog-to-digital converters;
projecting a second structured light to the scene in the field of view of the image sensor;
applying second reset signals to the pixel array to reset rows of the pixel array;
exposing the rows of the pixel array to the second structured light to cause second analog signals from the rows of the pixel array;
capturing a second structured-light image using the pixel array during a second frame period by converting the second analog signals from the rows of the pixel array into second digital outputs to form the second structured-light image using said one or more analog-to-digital converters;
applying third reset signals to the pixel array to reset the rows of the pixel array;
exposing the rows of the pixel array to non-structured light to cause third analog signals from the rows of the pixel array;
capturing the regular image using the pixel array during a third frame period by converting the third analog signals from the rows of the pixel array into third digital outputs to form the regular image using said one or more analog-to-digital converters;
deriving first depth information from the first structured-light image;
deriving second depth information from the second structured-light image; and
deriving current depth information associated with the regular image based on the first depth information and the second depth information; and
wherein same pixels of the pixel array are used for capturing the first structured-light image, the second structured-light image and the regular image;
wherein the regular image is captured before, after or between the first structured-light image and the second structured-light image; and
wherein the third frame period is longer than the first frame period and the second frame period.
2. The method of claim 1 , wherein when the regular image is captured between the first structured-light image and the second structured-light image, the current depth information is derived based on the first depth information and differences between the first depth information and the second depth information, or based on the second depth information and the differences between the first depth information and the second depth information.
3. The method of claim 1 , wherein the current depth information is derived row by row based on the first depth information and differences between the first depth information and the second depth information by taking into account of time elapsed between each corresponding rows in the first structured-light image and the regular image respectively, or based on the second depth information and the differences between the first depth information and the second depth information by taking into account of time elapsed between each corresponding rows in the second structured-light image and the regular image respectively.
4. The method of claim 1 , wherein a depth model is derived based on the first depth information and the second depth information, and the current depth information is derived using the depth model.
5. The method of claim 4 , wherein when the regular image is captured between the first structured-light image and the second structured-light image, the current depth information is interpolated using the depth model.
6. The method of claim 4 , wherein when the regular image is captured after the first structured-light image and the second structured-light image, the current depth information is predicted using the depth model.
7. A capsule camera system, comprising:
an image sensor comprising a pixel array being responsive to light energy received by the image sensor to produce pixel signals having a voltage level depending on the light energy received;
a structured light source;
a non-structured light source;
one or more output circuits coupled to the image sensor to access the pixel signals produced by the image sensor;
one or more analog-to-digital converters having a first dynamic range and a second dynamic range;
one or more timing and control circuits, wherein said one or more timing and control circuits are arranged to:
project a first structured-light to a scene in a field of view of the image sensor;
apply first reset signals to the pixel array to reset rows of the pixel array;
expose the rows of the pixel array to the first structured light to cause first analog signals from the rows of the pixel array;
capture a first structured-light image using the image sensor during a first frame period by converting the first analog signals from the rows of the pixel array into first digital outputs to form the first structured-light image using one or more analog-to-digital converters;
project a second structured light to the scene in the field of view of the image sensor;
apply second reset signals to the pixel array to reset rows of the pixel array;
expose the rows of the pixel array to the second structured light to cause second analog signals from the rows of the pixel array;
capture a second structured-light image using the image sensor during a second frame period by converting the second analog signals from the rows of the pixel array into second digital outputs to form the second structured-light image using said one or more analog-to-digital converters;
apply third reset signals to the pixel array to reset the rows of the pixel array;
exposing the rows of the pixel array to non-structured light to cause third analog signals from the rows of the pixel array, and
capture a regular image using the image sensor during a third frame period by converting the third analog signals from the rows of the pixel array into third digital outputs to form the regular image using said one or more analog-to-digital converters;
derive first depth information from the first structured-light image;
derive second depth information from the second structured-light image; and
derive current depth information associated with the regular image based on the first depth information and the second depth information; and
wherein same pixels of the pixel array are used for the first structured-light image, the second structured-light image and the regular image;
wherein the regular image is captured before, after or between the first structured-light image and the second structured-light image; and
wherein the third frame period is longer than the first frame period and the second frame period.