Associating LIDAR data and image data
A monocular image often does not contain enough information to determine, with certainty, the depth of an object in a scene reflected in the image. Combining image data and LIDAR data may enable determining a depth estimate of the object relative to the camera. Specifically, LIDAR points corresponding to a region of interest (“ROI”) in the image that corresponds to the object may be combined with the image data. These LIDAR points may be scored according to a monocular image model and/or a factor based on a distance between projections of the LIDAR points into the ROI and a center of the region of interest may improve the accuracy of the depth estimate. Using these scores as weights in a weighted median of the LIDAR points may improve the accuracy of the depth estimate, for example, by discerning between a detected object and an occluding object and/or background.
1. A method comprising:
receiving LIDAR data and image data associated with an environment;
determining, based at least in part on the image data, a region of interest identifying a portion of the image data as being associated with an object in the environment;
generating scores for LIDAR points associated with the region of interest, wherein generating a first score of the scores for a corresponding first LIDAR point is based, at least in part, on a distance from a first point in the image data within the region of interest to a second point in the image data associated with a projection of the first LIDAR point into the image data, and wherein the first score is non-binary;
determining, based at least in part on the scores, a weighted median of the LIDAR points; and
determining, based at least in part on the weighted median, a first depth estimate associated with a distance from a sensor to the object.
2. The method of claim 1 , wherein generating the first score comprises determining, by a machine-learned model, a probability distribution associated with a depth measurement associated with the first LIDAR point.
3. The method of claim 2 , wherein determining the probability distribution by the machine-learned model comprises:
providing at least one of an object detection or a classification associated with the object detection as input to the machine-learned model; and
receiving, from the machine-learned model, the probability distribution.
4. The method of claim 2 , further comprising:
determining, based at least in part on the probability distribution, a probability density associated with the depth measurement associated with the first LIDAR point; and
determining, based at least in part on a distance between the first LIDAR point projected into the image data and a center of the region of interest, a factor.
5. The method of claim 4 , wherein generating the first score associated with the first LIDAR point is based at least in part on the factor and the probability density.
6. The method of claim 1 , further comprising controlling an autonomous vehicle based at least in part on the first depth estimate.
7. An apparatus comprising:
one or more processors; and
a memory storing processor-executable instructions that, when executed by the one or more processors, cause the apparatus to perform operations comprising:
receiving LIDAR data and image data associated with an environment;
receiving a region of interest identifying a portion of the image data as being associated with an object in the environment;
generating scores for LIDAR points associated with the region of interest, wherein generating a first score for a corresponding first LIDAR point is based at least in part on a distance from a first point within the region of interest to a second point in the image data associated with the first LIDAR point, and wherein the first score increases or decreases based at least in part on distance; and
determining, based at least in part on the scores, a first depth estimate associated with a distance from a sensor to the object.
8. The apparatus of claim 7 , wherein generating the first score is based at least in part on:
providing at least one of an object detection or a classification associated with the object detection as input to a machine-learned model; and
receiving, from the machine-learned model, a probability distribution associating various depths with probabilities.
9. The apparatus of claim 8 , wherein generating the first score is based at least in part on a probability density and a factor, the operations further comprising:
determining, based at least in part on the probability distribution, the probability density associated with a depth measurement associated with the first LIDAR point; and
determining, based at least in part on a distance between the first LIDAR point projected into the image data and a center of the region of interest, the factor.
10. The apparatus of claim 7 , wherein determining the first depth estimate comprises:
sorting the LIDAR points by distance; and
determining, based at least in part on the sorted LIDAR points and the scores, a weighted median as the first depth estimate, wherein weights associated with the weighted median are based at least in part on the scores.
11. The apparatus of claim 7 , wherein the operations further comprise:
identifying a subset of LIDAR points associated with distance values outside a range of depth values that are based at least in part on the first depth estimate;
sorting the subset of LIDAR points by distances associated with the subset of LIDAR points;
determining, based at least in part on scores associated with the subset and the sorting, a second weighted median; and
identifying, as a secondary depth estimate, a depth measurement associated with the second weighted median.
12. The apparatus of claim 11 , wherein the operations further comprise identifying the secondary depth estimate as being associated with a second object in the environment.
13. The apparatus of claim 7 , wherein the operations further generating instructions for controlling an autonomous vehicle based at least in part on the first depth estimate.
14. The apparatus of claim 7 , wherein generating the first score comprises determining that a projection of the first LIDAR point is within the region of interest and determining the distance from the point within the region of interest to a projected point associated with the first LIDAR point.
15. A non-transitory computer-readable medium storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving LIDAR data and image data associated with an environment;
receiving a region of interest associated with an object in the environment;
generating scores for LIDAR points associated with the region of interest, wherein generating a first score for a corresponding first LIDAR point is based at least in part on a distance from a first point within the region of interest to a second point in the region of interest associated with the first LIDAR point, and wherein the first score increases or decreases based at least in part on distance; and
determining, based at least in part on the scores, a first depth estimate associated with a distance from a sensor to the object.
16. The non-transitory computer-readable medium of claim 15 , wherein generating the first score is based at least in part on:
providing at least one of a portion of image data associated with the region of interest or a classification associated with the portion of image data as input to a machine-learned model; and
receiving, from the machine-learned model, a probability distribution identifying at least a probability associated with a particular depth.
17. The non-transitory computer-readable medium of claim 16 , wherein generating the first score is based at least in part on a probability density and a factor, the operations further comprising:
determining, based at least in part on the probability distribution, the probability density associated with a depth measurement associated with the first LIDAR point; and
determining, based at least in part on a distance between the first LIDAR point projected into the image data and a center of the region of interest, the factor.
18. The non-transitory computer-readable medium of claim 15 , wherein determining the first depth estimate comprises:
sorting the LIDAR points by distance; and
determining, based at least in part on the sorted LIDAR points and the scores, a weighted median as the first depth estimate, wherein weights associated with the weighted median are based at least in part on the scores.
19. The non-transitory computer-readable medium of claim 15 , wherein the operations further comprise:
identifying a subset of LIDAR points associated with distance values outside a range of depth values that are based at least in part on the first depth estimate;
sorting the subset of LIDAR points by distances associated with the subset of LIDAR points;
determining, based at least in part on scores associated with the subset and the sorting, a second weighted median; and
identifying, as a secondary depth estimate, a depth measurement associated with the second weighted median.
20. The non-transitory computer-readable medium of claim 19 , wherein the operations further comprise identifying the secondary depth estimate as being associated with a second object in the environment.