Metrics for evaluating autonomous vehicle performance
Systems and methods for generating performance metrics for autonomous vehicle systems are provided. The performance metrics include two complementary metrics that evaluate a machine-learning object prediction model relative to a number of potential trajectories of an autonomous vehicle. The performance metrics include an avoidance metric that quantifies a probability that a region occupied by a real-world or simulated object is reached by the autonomous vehicle, the region is not blocked by another object, and the region is not blocked by a prediction output by the machine-learning object prediction model. The performance metrics also include an availability metric that quantifies a probability that a simulated or real-world object is not located within a region, the region is not blocked by another simulated or real-world object, and the autonomous vehicle is unnecessarily blocked by the prediction output before the autonomous vehicle reaches the particular footprint.
1 . An autonomous vehicle computing system for controlling an autonomous vehicle, the autonomous vehicle computing system comprising:
one or more processors;
a machine-learned prediction system having one or more parameters configured by:
(a) receiving data indicative of a trajectory of a test autonomous vehicle comprising a plurality of footprints of the test autonomous vehicle at a corresponding plurality of time steps of a time frame;
(b) receiving testing object data indicative of a ground truth object trajectory of a test object in a test environment of the test autonomous vehicle;
(c) receiving, from the machine-learned prediction system, prediction data indicative of one or more predicted object trajectories in the test environment;
(d) evaluating the machine-learned prediction system using a performance metric that characterizes protection of exposed ground truth occupancy, wherein the performance metric is based at least in part on, for a respective footprint of the plurality of footprints of the test autonomous vehicle:
(i) a probability that the ground truth object trajectory occupies at least a portion of the respective footprint at a respective time step corresponding to the respective footprint; and
(ii) a probability that the respective footprint is not blocked by at least one of the one or more predicted object trajectories before the respective time step; and
(e) updating, based at least in part on the evaluating of the machine-learned prediction system using the performance metric, the machine-learned prediction system to improve a value of the performance metric; and
one or more computer-readable medium storing instructions that when executed by the one or more processors cause the autonomous vehicle computing system to perform operations, the operations comprising:
processing sensor data descriptive of an object in an environment of the autonomous vehicle;
generating, using the machine-learned prediction system, one or more predictions for the object; and
controlling the autonomous vehicle based on the one or more predictions.
2 . The autonomous vehicle computing system of claim 1 , wherein (e) comprises:
training, using the performance metric as an objective, the machine-learned prediction system.
3 . The autonomous vehicle computing system of claim 1 , comprising a machine-learned motion planning system;
wherein (a) comprises processing, using the machine-learned motion planning system, the prediction data from the machine-learned prediction system to generate the trajectory; and
wherein controlling the autonomous vehicle based on the one or more predictions comprises:
generating a motion plan using the machine-learned motion planning system based on the one or more predictions for the object; and
controlling the autonomous vehicle based on to the motion plan.
4 . A computer-implemented method, comprising:
processing sensor data descriptive of an object in an environment of an autonomous vehicle;
generating, using a machine-learned prediction system of the autonomous vehicle, one or more predictions for the object, the machine-learned prediction system having one or more parameters configured by:
(a) receiving data indicative of a trajectory of a test autonomous vehicle comprising a plurality of footprints of the test autonomous vehicle at a corresponding plurality of time steps of a time frame;
(b) receiving testing object data indicative of a ground truth object trajectory of a test object in a test environment of the test autonomous vehicle;
(c) receiving, from the machine-learned prediction system, prediction data indicative of one or more predicted object trajectories in the test environment;
(d) evaluating the machine-learned prediction system using a performance metric that characterizes protection of exposed ground truth occupancy, wherein the performance metric is based at least in part on, for a respective footprint of the plurality of footprints of the test autonomous vehicle:
(i) a probability that the ground truth object trajectory occupies at least a portion of the respective footprint at a respective time step corresponding to the respective footprint; and
(ii) a probability that the respective footprint is not blocked by at least one of the one or more predicted object trajectories before the respective time step; and
(e) updating, based at least in part on the evaluating of the machine-learned prediction system using the performance metric, the machine-learned prediction system to improve a value of the performance metric; and
controlling the autonomous vehicle based on the one or more predictions.
5 . The computer-implemented method of claim 4 , wherein determining the probability that the respective footprint is not blocked by the at least one predicted object trajectory before the respective time step comprises:
determining a probability that the one or more predicted object trajectories do not overlap a preceding footprint at a preceding time step preceding the respective time step.
6 . The computer-implemented method of claim 4 , wherein the performance metric is based at least in part on a probability of the autonomous vehicle reaching the respective footprint during execution of the trajectory.
7 . The computer-implemented method of claim 4 , wherein the performance metric is based at least in part on a probability of the respective footprint not being blocked by another ground truth object at a preceding time step preceding the respective time step.
8 . The computer-implemented method of claim 4 , wherein the respective footprint comprises a plurality of vehicle cells within a spatio-temporal occupancy grid, wherein the spatio-temporal occupancy grid comprises a plurality of cells representative of the environment and the plurality of vehicle cells represent a region of fixed area within the environment corresponding to a body of the autonomous vehicle at the respective time step.
9 . The computer-implemented method of claim 8 , wherein the spatio-temporal occupancy grid is defined by path-relative coordinates adapted to an intended path of the autonomous vehicle, wherein the intended path of the autonomous vehicle is based at least in part on a road geometry associated with the environment.
10 . The computer-implemented method of claim 9 , wherein (a) comprises:
generating the data indicative of the trajectory of the autonomous vehicle based at least in part on the intended path of the autonomous vehicle; and
determining a trajectory probability for the trajectory, the trajectory probability indicative of a probability of the autonomous vehicle reaching the respective footprint during execution of the trajectory.
11 . The computer-implemented method of claim 4 , wherein the performance metric comprises an avoidance metric that correlates to recall of the ground truth object trajectories by the predicted object trajectories.
12 . The computer-implemented method of claim 4 , wherein (e) comprises:
training, using the performance metric as an objective, the machine-learned prediction system.
13 . The computer-implemented method of claim 4 , wherein (a) comprises:
generating, using a machine-learned motion planning system of the autonomous vehicle, the trajectory.
14 . The computer-implemented method of claim 13 , wherein the machine-learned motion planning system processes the prediction data from the machine-learned prediction system to generate the trajectory.
15 . The computer-implemented method of claim 14 , wherein the machine-learned prediction system processes sensor data descriptive of the environment.
16 . The computer-implemented method of claim 13 , wherein the machine-learned motion planning system generates a trajectory probability, the trajectory probability indicative of a probability of the autonomous vehicle reaching the respective footprint during execution of the trajectory.
17 . One or more non-transitory computer-readable media storing:
a machine-learned prediction system having one or more parameters configured by:
(a) receiving data indicative of a trajectory of a test autonomous vehicle comprising a plurality of footprints of the test autonomous vehicle at a corresponding plurality of time steps of a time frame;
(b) receiving testing object data indicative of a ground truth object trajectory of a test object in a test environment of the test autonomous vehicle;
(c) receiving, from the machine-learned prediction system, prediction data indicative of one or more predicted object trajectories in the test environment;
(d) evaluating the machine-learned prediction system using a performance metric that characterizes protection of exposed ground truth occupancy, wherein the performance metric is based at least in part on, for a respective footprint of the plurality of footprints of the test autonomous vehicle:
(i) a probability that the ground truth object trajectory occupies at least a portion of the respective footprint at a respective time step corresponding to the respective footprint; and
(ii) a probability that the respective footprint is not blocked by at least one of the one or more predicted object trajectories before the respective time step; and
(e) updating, based at least in part on the evaluating of the machine-learned prediction system using the performance metric, the machine-learned prediction system to improve a value of the performance metric; and
instructions that when executed by one or more processors cause a computing system to perform operations, the operations comprising:
processing sensor data descriptive of an object in an environment of an autonomous vehicle;
generating, using the machine-learned prediction system, one or more predictions for the object; and
controlling the autonomous vehicle based on the one or more predictions.
18 . The one or more non-transitory computer-readable media of claim 17 , wherein (e) comprises:
training, using the performance metric as an objective, the machine-learned prediction system.
19 . The one or more non-transitory computer-readable media of claim 17 , storing:
a machine-learned motion planning system;
wherein (a) comprises processing, using the machine-learned motion planning system, the prediction data from the machine-learned prediction system to generate the trajectory; and
wherein controlling the autonomous vehicle based on the one or more predictions comprises:
generating a motion plan using the machine-learned motion planning system based on the one or more predictions for the object; and
controlling the autonomous vehicle based on to the motion plan.
20 . The one or more non-transitory computer-readable media of claim 17 , wherein determining the probability that the respective footprint is not blocked by the at least one predicted object trajectory before the respective time step comprises:
determining a probability that the one or more predicted object trajectories do not overlap the respective footprint at the respective time step, and
determining a probability that the one or more predicted object trajectories do not overlap a preceding footprint at a preceding time step preceding the respective time step.