Method and apparatus for active depth sensing and calibration method thereof

1. A method of recovering a depth map from an observed infrared (IR) image using an active depth sensing device, the method comprising:

illuminating speckles in an IR spectrum by a projector of the active depth sensing device;

capturing speckles and synthesizing a horizon image, by an IR camera of the active depth sensing device, wherein the horizon image comprises a speckle pattern estimated by warping a plurality of images of varying depth, each of the plurality of images having a diverse set of pixel information in an x-direction from the IR camera to the projector, and the horizon image having a higher resolution in the x-direction than each of the plurality of images; and

estimating projector distortion based on speckle locations in the horizon image.

2. The method of claim 1 , wherein the IR camera is rigidly mounted with the projector in the active sensing device.

3. The method of claim 1 , wherein the projector distortion corresponds to a mapping between speckle coordinates of a binary pattern image and corresponding speckle coordinates of the horizon image.

4. The method of claim 3 , wherein the horizon image comprises a high resolution horizon image, and estimating the projector distortion comprises matching a brightest center speckle in the horizon image with a center speckle in the binary pattern image, and iteratively predicting speckle locations in the horizon image in areas bordering known speckles.

5. The method of claim 1 , further comprising:

estimating IR distortion in accordance with a focal length of the IR camera;

capturing, by the IR camera, speckles on the observed IR image;

determining speckle locations in the observed IR image in accordance with the projector distortion and the IR distortion; and

recovering the depth map from the observed IR image based on the determined speckle locations.

6. The method of claim 5 , wherein at least one of:

the focal length is proportional to an amount of shift a given depth induces between coordinates of the horizon image and coordinates of the observed IR image; and

the focal length maximizes a similarity between a pair of images warped to the horizon image using a focal length parameter.

7. The method of claim 5 , wherein determining speckle locations in the observed IR image comprises:

constructing a disparity map using the focal length;

inverting the disparity map to a perspective of the projector; and

predicting the speckle locations in the observed IR image using the speckle locations in the horizon image and the inverted disparity map.

8. The method of claim 7 , wherein constructing the disparity map comprises:

obtaining an initial disparity map;

cleaning the initial disparity map via median regularization; and

refining the cleaned disparity map via sub-pixel matching.

9. The method of claim 5 , wherein determining speckle locations in the observed IR image comprises:

modifying the speckle locations in the observed IR image in accordance with an intensity attenuation.

10. The method of claim 9 , wherein the intensity attenuation comprises a measured decrease in intensity of speckles in the IR camera as a distance from the IR camera to the speckles increases.

11. An electronic device, comprising:

a projector;

an infrared (IR) camera rigidly mounted with the projector;

a processor; and

a non-transitory computer readable storage medium storing instructions that, when executed, cause the processor to:

illuminate speckles in an IR spectrum by the projector;

capture, by the IR camera, speckles and synthesize a horizon image, wherein the horizon image comprises a speckle pattern estimated by warping a plurality of images of varying depth, the plurality of images having a diverse set of pixel information in an x-direction from the IR camera to the projector, and the horizon image having a higher resolution in the x-direction than each of the plurality of images; and

estimate projector distortion based on speckle locations in the horizon image.

12. The electronic device of claim 11 , wherein the projector distortion corresponds to a mapping between speckle coordinates of a binary pattern image and corresponding speckle coordinates of the horizon image.

13. The electronic device of claim 12 , wherein the horizon image comprises a high resolution horizon image, and estimating the projector distortion comprises matching a brightest center speckle in the horizon image with a center speckle in the binary pattern image, and iteratively predicting speckle locations in the horizon image in areas bordering known speckles.

14. The electronic device of claim 11 , wherein the non-transitory computer readable storage medium stores instructions that, when executed, further cause the processor to:

estimate IR distortion in accordance with a focal length of the IR camera;

capture, by the IR camera, speckles on the observed IR image;

determine speckle locations in the observed IR image in accordance with the projector distortion and the IR distortion; and

recover the depth map from the observed IR image based on the determined speckle locations.

15. The electronic device of claim 14 , wherein at least one of:

the focal length is proportional to an amount of shift a given depth induces between coordinates of the horizon image and coordinates of the observed IR image; and

the focal length maximizes a similarity between a pair of images warped to the horizon image using a focal length parameter.

16. The electronic device of claim 14 , wherein, in determining speckle locations in the observed IR image, the instructions further cause the processor to:

construct a disparity map using the focal length;

invert the disparity map to a perspective of the projector; and

predict the speckle locations in the observed IR image using the speckle locations in the horizon image and the inverted disparity map.

17. The electronic device of claim 14 , wherein, in constructing the disparity map, the instructions further cause the processor to:

obtain an initial disparity map;

clean the initial disparity map via median regularization; and

refine the cleaned disparity map via sub-pixel matching.

18. The electronic device of claim 14 , wherein determining speckle locations in the observed IR image comprises:

modifying the speckle locations in the observed IR image in accordance with an intensity attenuation comprising a measured decrease in intensity of speckles in the IR camera as a distance from the IR camera to the speckles increases.