System and method for active stereo depth sensing
An electronic device estimates a depth map of an environment based on stereo depth images captured by depth cameras having exposure times that are offset from each other in conjunction with illuminators pulsing illumination patterns into the environment. A processor of the electronic device matches small sections of the depth images from the cameras to each other and to corresponding patches of immediately preceding depth images (e.g., a spatio-temporal image patch “cube”). The processor computes a matching cost for each spatio-temporal image patch cube by converting each spatio-temporal image patch into binary codes and defining a cost function between two stereo image patches as the difference between the binary codes. The processor minimizes the matching cost to generate a disparity map, and optimizes the disparity map by rejecting outliers using a decision tree with learned pixel offsets and refining subpixels to generate a depth map of the environment.
1. A method comprising:
projecting, at an illuminator of an electronic device, a first illumination pattern into an environment of the electronic device at a frequency and a first phase and a second illumination pattern into the environment of the electronic device at the frequency and a second phase, wherein the second illumination pattern is rotated with respect to the first illumination pattern;
dynamically adjusting at least one of the first phase and the second phase;
capturing, at a depth camera of the electronic device, a first depth image of the environment at the frequency and the first phase and a second depth image of the environment at the frequency and the second phase;
computing, at a processor of the electronic device, a matching cost for corresponding portions of the first depth image and the second depth image, wherein each of the corresponding portions comprises a plurality of pixels of the first and second depth images; and
identifying the corresponding portions of the first depth image and the second depth image based on the matching cost to generate a disparity map indicating disparities between pixels of the corresponding portions of the first depth image and the second depth image; and
estimating a depth map of the environment based on the disparity map.
2. The method of claim 1 , wherein the second illumination pattern differs from the first illumination pattern.
3. The method of claim 1 , wherein the illuminator comprises a vertical cavity surface emitting laser (VCSEL).
4. The method of claim 1 , further comprising temporally varying the first and second illumination patterns.
5. The method of claim 1 , further comprising refining the disparities in the disparity map by identifying a probability that a pixel is valid and rejecting pixels with identified probabilities of validity below a threshold to estimate the depth map.
6. The method of claim 5 , wherein identifying a probability that a pixel is valid comprises identifying that the pixel is valid based on a decision tree that sparsely samples spatially neighboring pixels of the pixel.
7. The method of claim 1 , further comprising refining the disparities in the corresponding portions based on fitting a parabola to the disparities of each pixel.
8. The method of claim 1 , wherein computing the matching cost further comprises:
computing a matching cost for corresponding portions of the first depth image, the second depth image, and one or more depth images captured immediately prior to capturing the first depth image.
9. The method of claim 1 , further comprising:
tracking, using the electronic device, locations of a hand captured using the depth camera based on the depth map.
10. The method of claim 1 , wherein dynamically adjusting comprises dynamically adjusting at least one of the first phase and the second phase such that a time when the first illumination pattern is projected into the environment and a time when the second illumination pattern is projected into the environment are non-overlapping.
11. A method comprising:
pulsing, at a first illuminator of an electronic device, a first illumination pattern into an environment of the electronic device at a first phase and a first frequency;
pulsing, at a second illuminator of the electronic device, a second illumination pattern into the environment of the electronic device at a second phase and the first frequency, wherein the second illumination pattern is rotated with respect to the first illumination pattern;
dynamically adjusting at least one of the first phase and the second phase;
capturing, at a first depth camera of the electronic device, a first series of depth images of the environment at the first phase and the first frequency;
capturing, at a second depth camera of the electronic device, a second series of depth images of the environment at the second phase and the first frequency;
comparing, at a processor of the electronic device, a first patch of a first depth image of the first series of depth images to a second patch of a second depth image of the second series of depth images, wherein each patch comprises a plurality of pixels;
computing a cost function for the compared patches;
generating a disparity map indicating disparities between corresponding pixels of corresponding patches of the first depth image and the second depth image based on the cost function; and
refining the disparities of the disparity map to generate an estimated depth map of the environment.
12. The method of claim 11 , further comprising temporally varying the first and second illumination patterns.
13. The method of claim 11 , further comprising refining the disparities in the corresponding pixels by identifying a probability that a pixel is valid and rejecting pixels with identified probabilities of validity below a threshold.
14. The method of claim 11 , wherein identifying a probability that a pixel is invalid is based on a decision tree that sparsely samples spatially neighboring pixels of the pixel.
15. The method of claim 11 , wherein comparing further comprises:
comparing a first patch of a first depth image of the first series of depth images to a second patch of a second depth image of the second series of depth images and at least one patch of each of one or more depth images captured immediately prior to the first depth image.
16. The method of claim 11 , further comprising:
tracking, using the electronic device, locations of a hand captured using the first depth camera and the second depth camera based on the estimated depth map.
17. The method of claim 11 , wherein dynamically adjusting at least one of the first phase and the second phase comprises dynamically adjusting at least one of the first phase and the second phase such that a time when the first illumination pattern is pulsed into the environment and a time when the second illumination pattern is pulsed into the environment are non-overlapping.
18. An electronic device comprising:
a first illuminator to project a first illumination pattern into an environment at a first time;
a second illuminator to project a second illumination pattern into the environment at a second time different from the first time;
a first depth camera to capture a first depth image of the environment at the first time;
a second depth camera to capture a second depth image of the environment at the second time;
a processor to:
compute a matching cost for a spatio-temporal image patch, the spatio-temporal image patch generated from corresponding portions of the first depth image, the second depth image, and one or more depth images captured immediately prior to the first depth image, wherein each of the corresponding portions comprises a plurality of pixels of the depth images;
identify the corresponding portions of the first depth image and the second depth image that minimize the matching cost to generate a disparity map indicating disparities between pixels of the corresponding portions of the first depth image and the second depth image; and
refine the disparities of the disparity map to generate an estimated depth map of the environment.
19. The electronic device of claim 18 , wherein the second illumination pattern differs from the first illumination pattern.
20. The electronic device of claim 18 , wherein the second illumination pattern is rotated with respect to the first illumination pattern.
21. The electronic device of claim 18 , wherein the first and second illuminators are configured to temporally vary the first and second illumination patterns.
22. The electronic device of claim 18 , wherein the processor is to refine the disparities in the corresponding portions by identifying a probability that a pixel is valid and rejecting pixels with identified probabilities of validity below a threshold.
23. The electronic device of claim 22 , wherein the processor is to identify a probability that a pixel is invalid based on a decision tree that sparsely samples spatially neighboring pixels of the pixel.
24. The electronic device of claim 18 , wherein the processor is to refine the disparities in the corresponding portions based on fitting a parabola to the disparities of each pixel.
25. The electronic device of claim 18 , further comprising:
one or more additional illuminators, wherein each of the first illuminator, the second illuminator, and the one or more additional illuminators is mounted at a different angle.
26. The electronic device of claim 18 , wherein the processor is to track locations of a hand captured using the first depth camera and the second camera based on the estimated depth map.