Perception validation for autonomous vehicles
An example method includes (a) obtaining an object detection from a perception system that describes an object in an environment of the autonomous vehicle; (b) obtaining, from a reference dataset, a label that describes a reference position of the object in the environment; (c) determining a plurality of component divergence values respectively for a plurality of divergence metrics, wherein a respective divergence value characterizes a respective difference between the object detection and the label; (d) providing the plurality of component divergence values to a machine-learned model to generate a score that indicates an aggregate divergence between the object detection and the label, wherein the machine-learned model includes a plurality of learned parameters defining an influence of the plurality of component divergence values on the score; (e) evaluating a quality of a match between the object detection and the label based on the score.
1. A computer-implemented method for validating a perception system of an autonomous vehicle, comprising:
(a) obtaining an object detection from a perception system that describes an object in an environment of the autonomous vehicle;
(b) obtaining, from a reference dataset, a label that describes a reference position of the object in the environment;
(c) determining a plurality of component divergence values respectively for a plurality of divergence metrics, wherein a respective divergence value characterizes a respective difference between the object detection and the label;
(d) providing the plurality of component divergence values to a machine-learned model to generate a score that indicates an aggregate divergence between the object detection and the label, wherein the machine-learned model comprises a plurality of learned parameters defining an influence of the plurality of component divergence values on the score, wherein the score indicates a quality of a match between the object detection and the label; and
(e) validating the perception system based on the score.
2. The computer-implemented method of claim 1 , comprising:
assigning an evaluation state to a component of the perception system based on the score.
3. The computer-implemented method of claim 2 , comprising:
generating, using the machine-learned model, a respective score for a respective object detection of a plurality of object detections generated by the perception system to determine a quality of a match with a respective label corresponding to the respective object detection;
wherein assigning an evaluation state to the component of the perception system based on the score comprises:
determining a proportion of matches for the plurality of object detections that satisfy a threshold quality;
comparing the proportion to a target threshold proportion, and
assigning the evaluation state based on the comparison.
4. The computer-implemented method of claim 1 , comprising, prior to (c), determining the plurality of learned parameters by fitting a linear model to a plurality of unit tests, wherein a respective unit test comprises an example object detection and a unit test label indicating reference data for the object.
5. The computer-implemented method of claim 4 , wherein the respective unit test is stored in association with a ground truth designation indicating whether there is a material divergence between the example object detection and the unit test label.
6. The computer-implemented method of claim 1 , wherein:
the object detection and the label are obtained from a unit test associated with a positive match; and
(e) comprises:
determining that the score indicates a negative match based on a failure of the score to satisfy a threshold; and
updating one or more of the plurality of learned parameters to cause the score to satisfy the threshold.
7. The computer-implemented method of claim 1 , wherein the plurality of learned parameters respectively correspond to the plurality of divergence metrics.
8. The computer-implemented method of claim 1 , comprising:
weighting a respective contribution of the respective component divergence value using a context value obtained using a context metric, wherein the context value is based on an attribute of the object.
9. The computer-implemented method of claim 8 , comprising:
determining, using the context metric and based on an attribute of the object detection or the label, a context domain for the respective component divergence value; and
weighting the respective contribution of the respective component divergence value based on a weighting parameter associated with the context domain.
10. The computer-implemented method of claim 1 , wherein the score comprises a weighted combination of the plurality of component divergence values, wherein the plurality of learned parameters are used to perform the weighting in the weighted combination.
11. The computer-implemented method of claim 10 , wherein the weighted combination is a linear combination.
12. The computer-implemented method of claim 1 , wherein the score is generated using a piecewise function that is linear with respect to each component divergence metric.
13. The computer-implemented method of claim 12 , wherein the piecewise function comprises learnable parameters that are constrained from flipping a direction of a contribution of a component divergence metric to the score.
14. The computer-implemented method of claim 1 , wherein the plurality of divergence metrics are evaluated between a labeled bounding box and a detected bounding box from the object detection.
15. The computer-implemented method of claim 14 , wherein the plurality of divergence metrics comprise at least one of the following divergence metrics:
a two-dimensional intersection over union of the labeled bounding box and the detected bounding box;
a three-dimensional intersection over union of the labeled bounding box and the detected bounding box;
an intersection over union of a projection of the labeled bounding box into a range view and a projection of the detected bounding box into the range view;
a difference in volume between the labeled bounding box and the detected bounding box;
a difference between:
a detected distance between the labeled bounding box and a position associated with the autonomous vehicle, and
a detected distance between the detected bounding box and the position associated with the autonomous vehicle;
a difference between:
a predicted distance between an expected position of the labeled bounding box and a position associated with the autonomous vehicle, and
a predicted distance between an expected position of the detected bounding box and the position associated with the autonomous vehicle,
a difference in a detected forward velocity associated with the object and a labeled forward velocity associated with the object; or
a difference in a detected heading associated with the object and a labeled heading associated with the object.
16. The computer-implemented method of claim 1 , wherein:
the plurality of divergence metrics comprise a difference between:
a predicted distance between an expected position of the labeled bounding box and a position associated with the autonomous vehicle, and
a predicted distance between an expected position of the detected bounding box and the position associated with the autonomous vehicle; and
a respective contribution of a respective component divergence value using the difference is weighted based on a time horizon for which the predicted distances are obtained.
17. The computer-implemented method of claim 1 , wherein validating the perception system based on the score comprises comparing the score to a first threshold and a second threshold different than the first threshold, wherein:
a score satisfying the first threshold indicates that the perception system satisfactorily detected the object; and
a score satisfying the second threshold but not the first threshold indicates that the perception system suboptimally detected the object.
18. The computer-implemented method of claim 17 , comprising:
training, using a plurality of example matches having scores that satisfy the second threshold but not the first threshold, the perception system to improve an object detection performance; and
training, using a plurality of example matches having scores that do not satisfy the second threshold or the first threshold, the perception system to discard invalid object detections.
19. A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that are executable by the one or more processors to cause the computing system to perform operations, the operations comprising:
(a) obtaining an object detection from a perception system that describes an object in an environment of an autonomous vehicle;
(b) obtaining, from a reference dataset, a label that describes a reference position of the object in the environment;
(c) determining a plurality of component divergence values respectively for a plurality of divergence metrics, wherein a respective divergence value characterizes a respective difference between the object detection and the label;
(d) providing the plurality of component divergence values to a machine-learned model to generate a score that indicates an aggregate divergence between the object detection and the label, wherein the machine-learned model comprises a plurality of learned parameters defining an influence of the plurality of component divergence values on the score, wherein the score indicates a quality of a match between the object detection and the label; and
(e) validating the perception system based on the score.
20. An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising:
a perception system that is configured to parse input sensor data descriptive of an environment and output perception data tracking objects in the environment, wherein the perception system is validated by:
(a) obtaining a validation object detection from the perception system that describes a validation object in a validation environment;
(b) obtaining, from a reference dataset, a label that describes a reference position of the validation object in the validation environment;
(c) determining a plurality of component divergence values respectively for a plurality of divergence metrics, wherein a respective divergence value characterizes a respective difference between the validation object detection and the label;
(d) providing the plurality of component divergence values to a machine-learned model to generate a score that indicates an aggregate divergence between the validation object detection and the label, wherein the machine-learned model comprises a plurality of learned parameters defining an influence of the plurality of component divergence values on the score, wherein the score indicates a quality of a match between the object detection and the label; and
(e) validating the perception system based on the score.