Active stereo depth prediction based on coarse matching
An electronic device estimates a depth map of an environment based on matching reduced-resolution stereo depth images captured by depth cameras to generate a coarse disparity (depth) map. The electronic device downsamples depth images captured by the depth cameras and matches sections of the reduced-resolution images to each other to generate a coarse depth map. The electronic device upsamples the coarse depth map to a higher resolution and refines the upsampled depth map to generate a high-resolution depth map to support location-based functionality.
1. A method comprising:
projecting, at an illuminator of an electronic device, an illumination pattern into an environment of the electronic device;
capturing a first image of the environment from a first camera of an infrared stereo camera pair and a second image of the environment from a second camera of the infrared stereo camera pair;
downsampling the first image to generate a first reduced-resolution image and downsampling the second image to generate a second reduced-resolution image;
matching patches of the first reduced-resolution image to patches of the second reduced-resolution image to generate a first coarse depth map of the environment, wherein each patch comprises at least one pixel;
normalizing an intensity of each pixel based on neighboring pixels of each pixel;
upsampling the first coarse depth map to a resolution of the first image to generate a first full resolution depth map; and
employing the first full resolution depth map to support computer vision functionality at the electronic device.
2. The method of claim 1 , further comprising determining an error of the first full resolution depth map based on matching patches of the first full resolution depth map to patches of the first image based on color.
3. The method of claim 2 , further comprising:
matching patches of the second reduced-resolution image to patches of the first reduced-resolution image to generate a second coarse depth map of the environment; and
upsampling the second coarse depth map to a full resolution of the second image to generate a second full resolution depth map.
4. The method of claim 3 , further comprising:
determining an error of the second full resolution depth map based on matching patches of the second full resolution depth map to patches of the second image based on color.
5. The method of claim 3 , further comprising:
matching pixels of the first coarse depth map of the environment to pixels of the second coarse depth map of the environment to identify matching pixels having a disparity greater than a threshold; and
wherein determining the error comprises ignoring the identified matching pixels.
6. The method of claim 1 , further comprising adding details from the first image to the first full resolution depth map.
7. The method of claim 1 , wherein matching patches comprises:
assigning a value to each patch based on a contribution of each pixel of the patch, wherein the contribution is based on at least one of a distance of a pixel from a center pixel of the patch and a difference in color between the pixel and the center pixel of the patch; and
comparing the values of the patches of the first reduced-resolution image to the values of the patches of the second reduced-resolution image.
8. An electronic device comprising:
an illuminator to project an illumination pattern into an environment of the electronic device;
an infrared stereo camera pair comprising a first camera to capture a first image of the environment and a second camera to capture a second image of the environment; and
a processor configured to:
downsample the first image to generate a first reduced-resolution image and downsampling the second image to generate a second reduced-resolution image;
match patches of the first reduced-resolution image to patches of the second reduced-resolution image to generate a first coarse depth map of the environment, wherein each patch comprises one or more pixels;
normalize an intensity of each pixel based on neighboring pixels of each pixel;
upsample the first coarse depth map to a resolution of the first image to generate a first full resolution depth map; and
employ the first full resolution depth map to support computer vision functionality at the electronic device.
9. The electronic device of claim 8 , wherein the processor is further configured to determine an error of the first full resolution depth map based on matching patches of the first full resolution depth map to patches of the first image based on color.
10. The electronic device of claim 9 , wherein the processor is further configured to:
match patches of the second reduced-resolution image to patches of the first reduced-resolution image to generate a second coarse depth map of the environment; and
upsample the second coarse depth map to a full resolution of the second image to generate a second full resolution depth map.
11. The electronic device of claim 10 , wherein the processor is further configured to:
determine an error of the second full resolution depth map based on matching patches of the second full resolution depth map to patches of the second image based on color.