Calibration metrics for measuring trajectory prediction
The present technology pertains to measuring the quality of an uncertainty prediction provided by a trajectory prediction algorithm. An AI/ML platform can receive data including predicted trajectories of objects received from the trajectory prediction algorithm and observed paths for the objects received from the plurality of sensors. The predicted trajectories indicate predicted locations of the objects at a plurality of time intervals, and uncertainty predictions reflecting an uncertainty about the location of the objects at each of the predicted time intervals. The AI/ML platform can further determine respective standard deviations of the observed paths of the objects relative to the respective uncertainty predictions and plots the respective standard deviations against an ideal distribution of path distributions relative to the predicted locations of the objects at the plurality of time intervals.
1 . A method comprising:
receiving data including predicted trajectories of objects received from a trajectory prediction algorithm and observed paths for the objects, the predicted trajectories indicating predicted locations of the objects at a plurality of time intervals;
determining uncertainty predictions defining a predicted probabilistic distribution of potential object locations at each of the time intervals;
determining respective Mahalanobis distances, the Mahalanobis distances quantifying deviations of the observed paths of the objects from centers of the respective predicted probabilistic distributions relative to a spread defined by the uncertainty predictions;
plotting an actual distribution formed from the determined Mahalanobis distances against an ideal Rayleigh distribution, the ideal Rayleigh distribution representing an expected distribution of the Mahalanobis distances for calibrated uncertainty predictions;
determining, by comparing a characteristic of the actual distribution of Mahalanobis distances to a corresponding characteristic of the ideal Rayleigh distribution, that a calibration quality of the uncertainty predictions provided by the trajectory prediction algorithm indicates that the uncertainty predictions are underfitted or overfitted;
based on determining that the uncertainty predictions are underfitted or overfitted, adjusting a navigation path of an autonomous vehicle; and
controlling the autonomous vehicle along the adjusted navigation path.
2 . The method of claim 1 , further comprising:
adjusting the trajectory prediction algorithm to provide uncertainty predictions that better correlate to the ideal Rayleigh distribution.
3 . The method of claim 2 , wherein adjusting the trajectory prediction algorithm to provide uncertainty predictions that better correlate to the ideal Rayleigh distribution comprises:
training the trajectory prediction algorithm by providing additional training data pertaining to a class of scenarios.
4 . The method of claim 2 , wherein adjusting the trajectory prediction algorithm to provide uncertainty predictions that better correlate to the ideal Rayleigh distribution comprises:
training the trajectory prediction algorithm by adjusting loss values used to give feedback to a machine learning algorithm.
5 . The method of claim 1 , wherein the comparison between the actual distribution of Mahalanobis distances and the ideal Rayleigh distribution is performed independently for each of a plurality of directional components of the deviations.
6 . The method of claim 5 , wherein the plurality of directional components comprises at least one of a right-lateral component, a left-lateral component, a forward-longitudinal component, or rear-longitudinal component.
7 . An AI/ML platform for measuring quality of an uncertainty prediction provided by a trajectory prediction algorithm, comprising:
a storage configured to store instructions;
a processor configured to execute the instructions and cause the processor to:
receive data including predicted trajectories of objects received from a trajectory prediction algorithm and observed paths for the objects, the predicted trajectories indicating predicted locations of the objects at a plurality of time intervals;
determine uncertainty predictions defining a predicted probabilistic distribution of potential object locations at each of the time intervals;
determine respective Mahalanobis distances, the Mahalanobis distances quantifying deviations of the observed paths of the objects from centers of the respective predicted probabilistic distributions relative to a spread defined by the uncertainty predictions;
plot an actual distribution formed from the determined Mahalanobis distances against an ideal Rayleigh distribution, the ideal Rayleigh distribution representing an expected distribution of the Mahalanobis distances for calibrated uncertainty predictions;
determine, by comparing a characteristic of the actual distribution of Mahalanobis distances to a corresponding characteristic of the ideal Rayleigh distribution, that a calibration quality of the uncertainty predictions provided by the trajectory prediction algorithm indicates that the uncertainty predictions are underfitted or overfitted;
based on determining that the uncertainty predictions are underfitted or overfitted, adjust a navigation path of an autonomous vehicle; and
control the autonomous vehicle along the adjusted navigation path.
8 . The AI/ML platform of claim 7 , wherein the processor is configured to execute the instructions and cause the processor to:
adjust the trajectory prediction algorithm to provide uncertainty predictions that better correlate to the ideal Rayleigh distribution.
9 . The AI/ML platform of claim 8 , wherein the processor is configured to execute the instructions and cause the processor to:
train the trajectory prediction algorithm by providing additional training date-data pertaining to a class of scenarios.
10 . The AI/ML platform of claim 8 , wherein the processor is configured to execute the instructions and cause the processor to:
train the trajectory prediction algorithm by adjusting loss values used to give feedback to a machine learning algorithm.
11 . The AI/ML platform of claim 7 , wherein the comparison between the actual distribution of Mahalanobis distances and the ideal Rayleigh distribution is performed independently for each of a plurality of directional components of the deviations.
12 . The AI/ML platform of claim 11 , wherein the plurality of directional components comprises at least one of a right-lateral component, a left-lateral component, a forward-longitudinal component, or a rear-longitudinal component.
13 . A non-transitory computer readable medium comprising instructions, the instructions, when executed by a computing system, cause the computing system to:
receive data including predicted trajectories of objects received from a trajectory prediction algorithm and observed paths for the objects, the predicted trajectories indicating predicted locations of the objects at a plurality of time intervals;
determine uncertainty predictions at each of the time intervals defining a predicted probabilistic distribution of potential object locations;
determine respective Mahalanobis distances, the Mahalanobis distances quantifying deviations of the observed paths of the objects from centers of the respective predicted probabilistic distributions relative to a spread defined by the uncertainty predictions;
plot an actual distribution formed from the determined Mahalanobis distances against an ideal Rayleigh distribution, the ideal Rayleigh distribution representing an expected distribution of the Mahalanobis distances for calibrated uncertainty predictions;
determine, by comparing a characteristic of the actual distribution of Mahalanobis distances to a corresponding characteristic of the ideal Rayleigh distribution, that a calibration quality of the uncertainty predictions provided by the trajectory prediction algorithm indicates that the uncertainty predictions are underfitted or overfitted;
based on determining that the uncertainty predictions are underfitted or overfitted, adjust a navigation path of an autonomous vehicle; and
control the autonomous vehicle along the adjusted navigation path.
14 . The computer readable medium of claim 13 , wherein the computer readable medium further comprises instructions that, when executed by the computing system, cause the computing system to:
adjust the trajectory prediction algorithm to provide uncertainty predictions that better correlate to the ideal Rayleigh distribution.
15 . The computer readable medium of claim 14 , wherein the computer readable medium further comprises instructions that, when executed by the computing system, cause the computing system to:
train the trajectory prediction algorithm by providing additional training date-data pertaining to a class of scenarios.
16 . The computer readable medium of claim 14 , wherein the computer readable medium further comprises instructions that, when executed by the computing system, cause the computing system to:
train the trajectory prediction algorithm by adjusting loss values used to give feedback to a machine learning algorithm.
17 . The computer readable medium of claim 13 , wherein the comparison between the actual distribution of Mahalanobis distances and the ideal Rayleigh distribution is performed independently for each of a plurality of directional components of the deviations.