Apparatus and methods for three-dimensional sensing

1. A method for sensing depth, the method comprising:

using at least two dual band near-infrared (NIR) to simultaneously capture one or more images;

alternatively, simultaneously capturing a first pair of the one or more images captured with a pattern IR illuminator in an off state and simultaneously a second pair of the one or more images captured with the pattern IR illuminator in an on state;

generating a disparate stream from the second pair of the one or more images;

performing stereo depth to the second pair of the one or more images by:

selecting a point in one image of the second pair of the one or more images;

computing, by a computing processor, a matching score along an epipolar line in another of the second pair of the captured images;

minimizing the matching score to thereby identify disparity; and

performing a shift-to-depth based on geometric and optical properties of the one image and the another image of the second pair of the images.

2. The method of claim 1 , further comprising:

performing image processing to correct one or more of optical distortion, rectification, dark image bias, lens shadow, and hot pixels of the simultaneously captured first pair of the one or more images.

3. The method of claim 1 , further comprising:

applying image processing to the simultaneously captured first pair of the one or more images to enhance signal-to-noise ratio.

4. A method for sensing depth, the method comprising:

alternatively, capturing by two dual band cameras a first set of frames and a second set of frames, the first set of frames forming a stereo pair captured with a pattern IR illuminator in an off state and the second set of frames having stereo pairs captured with the pattern IR illuminator in an on state;

generating a disparate stream from the second set of frames;

performing stereo to the second set of frames by:

selecting a point in a first frame of the second set of frames;

computing, by a computer processor, a matching score along an epipolar line in a second frame of the second set of frames;

minimizing the matching score to thereby identify disparity; and

performing a shift-to-depth based on geometric and optical properties of the second set of frames.

5. The method of claim 4 , further comprising:

performing image processing to correct one or more of optical distortion, rectification, dark image bias, lens shadow, and hot pixels of the first set of frames and the second set of frames.

6. The method of claim 4 , further comprising:

applying image processing to the first set of frames and the second set of frames to enhance signal-to-noise ratio.

7. The method of claim 4 , wherein performing stereo further includes:

using a heterogeneous computing processor or graphical computing processor to compute another matching score on a lower resolution than a resolution of the matching score, and

refining an upsampled image on a graphical computing processor.

8. The method of claim 4 , wherein performing stereo further includes:

computing another matching score with respect to an output disparity resolution, wherein the another matching score is computed using lower resolutions for patch matching on a graphical computing processor.

9. The method of claim 4 , wherein the computing processor is a graphical computing processor.

10. The method of claim 4 , further comprising determining an illumination pattern at a higher signal-to-noise ratio for the second set of frames.

11. The method of claim 4 , wherein the second set of frames are used to render a stereo view of a world for mixed reality.

12. A method for sensing depth, the method comprising:

capturing a first image by a near-infrared (NIR) camera of an image device;

substantially simultaneously with capturing the first image, capturing a second image by an ultraviolet (UV) camera of the image device, the NIR camera and the UV camera aligned horizontally along the image device with an IR emitter of the image device and the first image and second image captured while the IR emitter is in an on state such that the first image and second image form a pattern stereo pair;

capturing a third image the NIR camera of an image device;

substantially simultaneously with capturing the third image, capturing a fourth image by the UV camera while the IR emitter is in an off state such that the third image and forth image form a visible stereo pair;

generating a first disparate stream from the first image and second image;

generating a second disparate stream from the third image and fourth image;

performing stereo depth computation on the first disparate stream by:

selecting a pixel in the first image;

determining an epipolar line between the first image and the second image;

computing, by a computer processor, a matching score of a pixel in the second image along the epipolar line;

identifying a disparity between the first image and the second image based at least in part on the matching score; and

performing a shift-to-depth based on geometric and optical properties of the first image and the second image in response to identifying the disparity.

13. The method of claim 12 , wherein:

the IR emitter is positioned between the NIR camera and the UV camera, and the NIR camera has a baseline closer to the emitter than the UV camera.

14. The method of claim 12 , further comprising selecting the first image, prior to selecting a pixel in the first image, based at least in part on one or more of a receive signal of signal-to-noise ratio, wherein the one or more of a receive signal of signal-to-noise ratio is determined based at least in part on pattern knowledge and information on scene distance from the first image or a prior image.

15. A method of performing depth sensing using multi-purpose camera modules, the method comprising:

using at least two dual band near-infrared (NIR) or ultraviolet (UV) cameras to simultaneously capture one or more stereo pairs, wherein the at least two dual band NIR or UV cameras are configured to capture images using visible light and NIR or UV;

alternating between two types of stereo pairs using the at least two dual band NIR or UV cameras by:

capturing a first stereo pair with a pattern IR illuminator in an off state (a “visible stereo pair”), and

capturing a second stereo pair with the pattern IR illuminator in an on state (a “pattern stereo pair”);

generating two disparate streams produced by steps (i)-(ii); and

performing stereo depth computation on frames captured with the pattern IR illuminator in the on state by:

(a) selecting a point in one image of the simultaneously captured one or more images,

(b) computing, by a computer processor, a matching score along an epipolar line in another of the simultaneously captured images,

(c) minimizing the matching score to thereby identify disparity, and

(d) performing a shift-to-depth based on geometric and optical properties of the one image and the another image of the simultaneously captured images.

16. The method of claim 4 , wherein capturing the first set of frames is performed using diffuse light-emitting diode (LED) illuminators in an on state while the pattern infrared (IR) illuminator is in the off state, and

wherein capturing the second frame is performed using diffuse LED illuminators in an on state while the pattern IR illuminator is in the on state.

17. The method of claim 16 , further comprising:

determining an illumination pattern at a higher signal-to-noise ratio for the second set of frames.

18. The method of claim 16 , wherein the second set of frames are used to render a stereo view of a world for mixed reality.

19. The method of claim 16 , wherein the first set of frames are used for six degree of freedom positional tracking using sparse feature points where a three-dimensional position of the feature points are triangulated using optical properties of the at least two dual band cameras and a geometric relationship between the at least two dual band cameras.

20. The method of claim 16 , wherein computation for stereo depth, positional tracking, and gesture/controller tracking is performed on a host processor from the at least two dual band cameras to another processor via USB for computation.

21. The method of claim 16 wherein the first set of frames are used to render a stereo view of a world for mixed reality.

22. The method of claim 16 , wherein computation for stereo depth, positional tracking, and gesture/controller tracking is performed on a host processor or the host processor streams data from the at least two dual band cameras to another processor via USB for computation.

23. The method of claim 16 , wherein the first set of frames are used to track hand gestures and position.

24. The method of claim 15 , further comprising performing image processing to correct one or more of optical distortion, rectification, dark image bias, lens shadow, and hot pixels of the one or more stereo pairs.

25. The method of claim 15 , further comprising applying image processing to the one or more stereo pairs to enhance signal-to-noise ratio.

26. The method of claim 15 , wherein the pattern IR illuminator is positioned between the at least two dual band near-infrared (NIR) or ultraviolet (UV) cameras.

27. A method of performing depth sensing using multi-purpose camera modules, the method comprising:

using at least two dual band cameras to simultaneously capture a plurality of frames, each of the plurality of frames including stereo pair of images, wherein the at least two dual band cameras are configured to capture images using visible light and near-infrared (NIR);

alternatively, capturing a first set of frames and a second set of frames, the first set of frames having a stereo pair captured with a pattern IR illuminator in an off state and the second set of frames having a stereo pairs captured with the pattern IR illuminator in an on state;

generating a first disparate stream from the first set of frames;

generating a second disparate stream from the second set of frames;

performing stereo depth computation on the second set of frames by:

selecting a point in a first image of individual stereo pairs of the second set of frames, computing, by a computer processor, a matching score along an epipolar line in a second image of the individual stereo pairs of the second set of frames,

reducing disparities between the first image and the second image of the individual stereo pairs of the second set of frames based at least in part on the matching score, and

performing a shift-to-depth based on geometric and optical properties of the first image and the second image of the individual stereo pairs of the second set of frames.

28. The method of claim 27 , wherein the second set of frames are used for six degree of freedom positional tracking using sparse feature points where a three-dimensional position of the feature points are triangulated using a dense image based tracking method.

29. The method of claim 27 , wherein the at least two dual band cameras are near-infrared (NIR) cameras.

30. The method of claim 27 , wherein the at least two dual band cameras are ultraviolet (UV) cameras.

31. The method of claim 16 , wherein the first set of frames are used to track a controller that has IR LEDs.