Image-enhanced depth sensing using machine learning

1. A method of training a neural network for depth computation, the method comprising:

providing an image input to the neural network, the image input including a camera image of a training scene;

providing a depth input to the neural network, the depth input including:

a sparse depth map of the training scene, wherein the sparse depth map is generated by:

for each of a plurality of sampling points defined by a sampling mask, setting the sparse depth map equal to a high-density depth map of the training scene generated using a high-density depth sensor; and

for each of a plurality of remaining points defined by the sampling mask, setting the sparse depth map equal to the high-density depth map at a nearest point of the plurality of sampling points; and

a distance map of the training scene, wherein the distance map is generated by:

for each of the plurality of sampling points, setting the distance map equal to zero; and

for each of the plurality of remaining points, setting the distance map equal to a distance from the nearest point of the plurality of sampling points;

generating, using the neural network, a computed depth map of the training scene based on the image input and the depth input;

computing an error between the computed depth map and the high-density depth map; and

modifying the neural network based on the error.

2. A method of training a neural network for depth computation, the method comprising:

providing a training image input to the neural network, the training image input including a camera image of a training scene;

obtaining a high-density depth map of the training scene captured using a high-density depth sensor;

sampling the high-density depth map to produce a sampled version of the high-density depth map;

providing a training depth input to the neural network, the training depth input based at least in part on the sampled version of the high-density depth map;

generating, using the neural network, a computed depth map of the training scene based on the training image input and the training depth input; and

modifying the neural network based on an error between the computed depth map of the training scene and the high-density depth map;

wherein, as a result of training the neural network, the neural network is configured to generate a computed depth map of a runtime scene based on the neural network being provided (i) a runtime image input including a camera image of the runtime scene captured using a camera and (ii) a runtime depth input based on a low-density depth map of the runtime scene captured using a low-density depth sensor separate from the camera.

3. The method of claim 2 , further comprising:

capturing, using the camera, the camera image of the training scene; and

capturing, using the high-density depth sensor, the high-density depth map.

4. The method of claim 2 , wherein the camera image of the training scene is a grayscale image having dimensions of H×W×1.

5. The method of claim 2 , wherein the camera image of the training scene is an RGB image having dimensions of H×W×3.

6. The method of claim 2 , wherein the training depth input includes a sparse depth map generated based on the sampled version of the high-density depth map and a sampling mask that indicates a plurality of sampling points used for sampling the high-density depth map.

7. The method of claim 6 , wherein the sparse depth map is generated by:

for each of the plurality of sampling points, setting the sparse depth map equal to the high-density depth map; and

for each of a plurality of remaining points, setting the sparse depth map equal to the high-density depth map at a nearest point of the plurality of sampling points.

8. The method of claim 6 , wherein the training depth input further includes a distance map generated by:

for each of the plurality of sampling points, setting the distance map equal to zero; and

for each of a plurality of remaining points, setting the distance map equal to a distance from a nearest point of the plurality of sampling points.

9. A method of using a neural network for depth computation, the method comprising:

capturing, using a camera, a camera image of a runtime scene;

capturing, using a low-density depth sensor, a low-density depth map of the runtime scene, the low-density depth sensor being separate from the camera;

providing a runtime image input to the neural network, the runtime image input including the camera image of the runtime scene;

providing a runtime depth input to the neural network, the runtime depth input based at least in part on the low-density depth map of the runtime scene, wherein the runtime depth input includes a sparse depth map of the runtime scene generated by:

for each of a plurality of sampling points determined by the low-density depth map of the runtime scene, setting the sparse depth map of the runtime scene equal to the low-density depth map of the runtime scene; and

generating, using the neural network, a computed depth map of the runtime scene based on the neural network being provided the runtime image input and the runtime depth input.

10. The method of claim 9 , wherein the sparse depth map of the runtime scene is further generated by:

for each of a plurality of remaining points, setting the sparse depth map of the runtime scene equal to the low-density depth map of the runtime scene at a nearest point of the plurality of sampling points.

11. The method of claim 9 , wherein the runtime depth input includes a distance map of the runtime scene generated by:

for each of a plurality of sampling points determined by the low-density depth map of the runtime scene, setting the distance map of the runtime scene equal to zero; and

for each of the plurality of remaining points, setting the distance map of the runtime scene equal to a distance from the nearest point of the plurality of sampling points.

12. The method of claim 9 , wherein the neural network was previously trained by:

providing a training image input to the neural network, the training image input including a camera image of a training scene;

providing a training depth input to the neural network, the training depth input based at least in part on a high-density depth map of the training scene;

generating, using the neural network, a computed depth map of the training scene; and

modifying the neural network based on an error between the computed depth map of the training scene and the high-density depth map of the training scene.

13. The method of claim 12 , further comprising:

generating, using the neural network, the computed depth map based on the training image input and the training depth input.

14. The method of claim 12 , further comprising:

capturing, using the camera, the camera image of the training scene; and

capturing, using a high-density depth sensor, the high-density depth map of the training scene.

15. The method of claim 12 , wherein the camera image of the training scene is a grayscale image having dimensions of H×W×1.

16. The method of claim 12 , wherein the camera image of the training scene is an RGB image having dimensions of H×W×3.

17. The method of claim 12 , wherein the training depth input includes a sparse depth map of the training scene generated based on:

the high-density depth map; and

a sampling mask that indicates a plurality of sampling points of the low-density depth sensor.

18. The method of claim 17 , wherein the sparse depth map of the training scene is generated by:

for each of the plurality of sampling points, setting the sparse depth map of the training scene equal to the high-density depth map; and

for each of a plurality of remaining points, setting the sparse depth map of the training scene equal to the high-density depth map at a nearest point of the plurality of sampling points.

19. The method of claim 17 , wherein the training depth input further includes a distance map of the training scene generated by:

for each of the plurality of sampling points, setting the distance map of the training scene equal to zero; and

for each of a plurality of remaining points, setting the distance map of the training scene equal to a distance from a nearest point of the plurality of sampling points.