Sensor fusion for depth estimation

1. A method for calculating a depth value for a pixel in a reference image, the method comprising:

receiving the reference image captured by a reference camera and at least one auxiliary image captured by an auxiliary camera;

generating a first support value indicating whether the pixel in the reference image is at a particular depth, relative to the reference camera, based on comparing a region of the auxiliary image captured by the auxiliary camera with a region of the reference image captured by the reference camera;

providing a depth estimate of the pixel from a range-estimation camera;

generating a second support value indicating whether the pixel in the reference image is at the particular depth based on comparing the depth estimate from the range-estimation camera to the particular depth;

generating a third support value indicating whether the pixel is at the particular depth based on projecting a 3D point, corresponding to the pixel in the reference image, onto the auxiliary image; and

fusing, by operation of one or more computer processors, the first, second, and third support values to generate a total support value for the pixel at the particular depth.

2. The method of claim 1 , further comprising:

calculating respective first, second, and third support values for the pixel at each of a plurality of discrete depths that divide a depth volume; and

fusing the respective first, second, and third support values to generate a respective total support value for each of the plurality of discrete depths; and

selecting the depth value for the pixel based on the discrete depth associated with the greatest respective total support value.

3. The method of claim 2 , further comprising:

selecting respective depth values for a plurality of pixels in the reference image based on calculating first, second, and third support values at each of the plurality of discrete depths; and

generating a depth map associated with the reference image using the respective depth values.

4. The method of claim 1 , wherein generating the first support value comprises:

projecting the auxiliary image, at the particular depth, onto the reference image; and

determining the first support value based on comparing a feature of a plurality of pixels in the reference image to a feature of a corresponding plurality of pixels in the reference image.

5. The method of claim 1 , wherein generating the third support value comprises:

determining the third support value by comparing the color of the pixel in the reference image with a corresponding pixel in the auxiliary image.

6. The method of claim 1 , further comprising:

generating a depth map by selecting respective depth values for a plurality of pixels in the image;

forming a super pixel comprising a plurality of pixels in the reference image based on features extracted from the reference image and a corresponding thermal image;

determining a depth-plane of the super pixel based on the depth values of the plurality of pixels; and

adjusting at least one of the depth values based on the depth-plane.

7. The method of claim 1 , wherein the reference camera, auxiliary camera, and range-estimation camera are mounted on a common support structure.

8. A computer program product for calculating a depth value for a pixel in a reference image, the computer program product comprising:

a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code configured to:

receive the reference image captured by a reference camera and at least one auxiliary image captured by an auxiliary camera;

generate a first support value indicating whether the pixel in the reference image is at a particular depth, relative to the reference camera, based on comparing a region of the auxiliary image with a region of the reference image;

provide a depth estimate of the pixel from a range-estimation camera;

generate a second support value indicating whether the pixel in the reference image is at the particular depth based on comparing the depth estimate from the range-estimation camera to the particular depth;

generate a third support value indicating whether the pixel is at the particular depth based on projecting a 3D point, corresponding to the pixel in the reference image, onto the auxiliary image; and

fuse the first, second, and third support values to generate a total support value for the pixel at the particular depth.

9. The computer program product of claim 8 , wherein the computer-readable program code is further configured to:

calculate respective first, second, and third support values for the pixel for each of a plurality of discrete depths that divide a depth volume; and

fuse the respective first, second, and third support values to generate a respective total support value for each of the plurality of discrete depths; and

select the depth value for the pixel based on the discrete depth associated with the greatest respective total support value.

10. The computer program product of claim 9 , wherein the computer-readable program code is further configured to:

select respective depth values for a plurality of pixels in the reference image based on calculating first, second, and third support values at each of the plurality of discrete depths; and

generate a depth map associated with the reference image using the respective depth values.

11. The computer program product of claim 8 , wherein generating the first support value comprises computer-readable program code further configured to:

project the auxiliary image, at the particular depth, onto the reference image; and

determine the first support value based on comparing a feature of a plurality of pixels in the reference image to a feature of a corresponding plurality of pixels in the reference image.

12. The computer program product of claim 8 , wherein generating the third support value comprises computer-readable program code further configured to:

determine the third support value by comparing the color of the pixel in the reference image with a corresponding pixel in the auxiliary image.

13. The computer program product of claim 8 , wherein the computer-readable program code is further configured to:

generate a depth map by selecting respective depth values for a plurality of pixels in the image;

form a super pixel comprising a plurality of pixels in the reference image based on features extracted from the reference image and a corresponding thermal image;

determine a depth-plane of the super pixel based on the depth values of the plurality of pixels; and

adjust at least one of the depth values based on the depth-plane.

14. A system, comprising:

a common support structure comprising:

a reference camera,

at least one auxiliary camera, and

a range-estimation camera; and

a computing device communicatively coupled to the reference, auxiliary, and range-estimation cameras, the computing device configured to:

calculate a first support value indicating whether a pixel in a reference image captured by the reference camera is at a particular depth, relative to the reference camera, based on comparing a region of the reference image with a region of an auxiliary image captured by the auxiliary camera;

calculate a second support value indicating whether the pixel in the reference image is at the particular depth based on comparing a depth estimate from the range-estimation camera to the particular depth;

calculate a third support value indicating whether the pixel is at the particular depth based on projecting a 3D point, corresponding to the pixel in the reference image, onto the auxiliary image; and

fuse the first, second, and third support values to generate a total support value for the pixel at the particular depth.

15. The system of claim 14 , wherein the computing device is further configured to:

calculate respective first, second, and third support values for the pixel for each of a plurality of discrete depths that divide a depth volume;

fuse the respective first, second, and third support values to generate a respective total support value for each of the plurality of discrete depths; and

select the depth value for the pixel based on the discrete depth associated with the greatest respective total support value.

16. The system of claim 15 , wherein the computing device is further configured to:

select respective depth values for a plurality of pixels in the image based on calculating first, second, and third support values at each of the plurality of discrete depths; and

generate a depth map associated with the image using the respective depth values.

17. The system of claim 14 , wherein the support structure includes at least two auxiliary cameras, wherein a first one of the auxiliary cameras is positioned on a first side of the reference camera and a second one of the auxiliary cameras is positioned on a second side of the reference camera opposite of the first side.

18. The system of claim 14 , wherein the support structure includes a thermal camera, wherein the thermal camera is configured to provide thermal data associated with the image to the computing device.

19. The system of claim 14 , wherein calculating the first support value comprises:

projecting the auxiliary image, at the particular depth, onto the reference image; and

determining the first support value based on comparing a feature of a plurality of pixels in the reference image to a feature of a corresponding plurality of pixels in the reference image.

20. The system of claim 14 , wherein calculating the third support value comprises:

determining the third support value by comparing the color of the pixel in the reference image with a corresponding pixel in the auxiliary image.