Dense depth estimation of image data
Systems, devices, and methods are described for generating dense depth estimates, and confidence values associated with such depth estimates, from image data. A machine learning algorithm can be trained using image data and associated depth values captured by one or more LIDAR sensors providing a ground truth. When the algorithm is deployed in a machine vision system, image data and/or depth data can be used to determine dense depth estimates for all pixels of the image data, as well as confidence values for each depth estimate. Such confidence values may be indicative of how confident the machine learned algorithm is of the associated depth estimate.
1. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the system to perform operations comprising:
receiving image data captured by an image sensor;
inputting at least a portion of the image data to a machine learned algorithm;
receiving, from the machine learned algorithm, a depth estimate associated with a pixel of the image data, the depth estimate indicative of a distance between a location associated with the image sensor and a surface represented by the pixel, wherein the machine learned algorithm comprises a convolutional neural network trained based at least in part on:
training image data comprising a first plurality of data points; and
ground truth data associated with the training image data, the ground truth data comprising a second plurality of data points, wherein the second plurality of data points comprises fewer data points than the first plurality of data points; and
receiving, from the machine learned algorithm, a confidence value associated with the depth estimate.
2. The system of claim 1 , the operations further comprising:
receiving depth data captured by a lidar sensor; and
inputting the depth data to the machine learned algorithm, wherein the depth estimate associated with the pixel is based at least in part on the depth data.
3. The system of claim 2 , the operations further comprising:
receiving a depth value indicator indicating whether the depth data exists for the pixel,
wherein the depth value indicator is based at least in part on a transformation of the depth data into a reference frame of the image data.
4. The system of claim 2 , wherein the depth data is a sparse data set relative to a number of pixels associated with the image data.
5. The system of claim 1 , wherein at least one of the image data, the training image data, or the ground truth data comprises simulated data.
6. The system of claim 1 , the operations further comprising:
determining the depth estimate based at least in part on a rectified linear unit (ReLU) activation function; and
determining the confidence value based at least in part on a sigmoid activation function.
7. A method comprising:
receiving image data captured by an image sensor;
inputting at least a portion of the image data to a machine learned algorithm;
receiving, from the machine learned algorithm, a depth estimate associated with a pixel of the image data, the depth estimate indicative of a distance between a first point associated with the image sensor and a surface represented by the pixel, wherein the algorithm is a machine learned algorithm trained based at least in part on:
training image data comprising a first plurality of data points; and
ground truth data associated with the training image data, the ground truth data comprising a second plurality of data points; and
receiving, from the machine learned algorithm, a confidence value associated with the depth estimate.
8. The method of claim 7 , further comprising:
providing the depth estimate and the confidence value to at least one of a perception system or a planning system of a vehicle.
9. The method of claim 8 , further comprising:
determining the depth estimate based at least in part on a rectified linear unit (ReLU) activation function; and
determining the confidence value based at least in part on a sigmoid activation function.
10. The method of claim 7 , further comprising:
receiving depth data from a lidar sensor; and
inputting the depth data to the machine learned algorithm, wherein the depth estimate associated with the pixel based at least in part on the depth data.
11. The method of claim 10 , further comprising:
receiving a depth value indicative of whether the depth data exists for the pixel,
wherein the depth value is based at least in part on a transformation of the depth data into a reference frame of the image data.
12. The method of claim 10 , wherein the depth data is a sparse data set relative to a number of pixels associated with the image data.
13. The method of claim 7 , further comprising:
determining depth estimates associated with a plurality of pixels; and
determining a plurality of confidence values associated with the depth estimates.
14. One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising:
receiving image data captured by an image sensor;
inputting at least a portion of the image data to a machine learned algorithm;
receiving, from the machine learned algorithm, a depth estimate associated with an image element of the image data, the depth estimate indicative of a distance between a first point associated with the image sensor and a surface represented by the image element, wherein the machine learned algorithm comprises a convolutional neural network trained based at least in part on:
training image data comprising a first plurality of data points; and
ground truth data associated with the training image data, the ground truth data comprising a second plurality of data points; and
receiving, from the machine learned algorithm, a confidence value associated with the depth estimate.
15. The one or more non-transitory computer-readable media of claim 14 , the operations further comprising:
receiving depth data from a lidar sensor; and
inputting the depth data to the machine learned algorithm substantially simultaneously with the image element, wherein the depth estimate associated with the image element based at least in part on the depth data.
16. The one or more non-transitory computer-readable media of claim 15 , the operations further comprising:
determining a depth value indicative of whether the depth data exists for the image element,
wherein the depth value is based at least in part on a transformation of the depth data into a reference frame of the image data.
17. The one or more non-transitory computer-readable media of claim 15 , wherein the depth data is a sparse data set relative to a number of image elements associated with the image data.