Probabilistic prediction of dynamic object behavior for autonomous vehicles
Systems and methods are described that probabilistically predict dynamic object behavior. In particular, in contrast to existing systems which attempt to predict object trajectories directly (e.g., directly predict a specific sequence of well-defined states), a probabilistic approach is instead leveraged that predicts discrete probability distributions over object state at each of a plurality of time steps. In one example, systems and methods predict future states of dynamic objects (e.g., pedestrians) such that an autonomous vehicle can plan safer actions/movement.
1. An object motion prediction system, the object motion prediction system comprising:
one or more processors; and
one or more non-transitory computer-readable media that store:
a machine-learned object motion prediction model configured to generate probabilistic predictions for object motion over a plurality of time steps, wherein the machine-learned object motion prediction model comprises a backbone portion and a prediction head; and
instructions that are executable by the one or more processors to cause the object motion prediction system to perform operations, the operations comprising:
obtaining a scene rasterization that describes a scene that includes a dynamic object;
processing the scene rasterization with the backbone portion of the machine-learned object motion prediction model to generate a feature map; and
processing the feature map with the prediction head of the machine-learned object motion prediction model to generate a plurality of probabilistic predictions respectively for the plurality of time steps, wherein the probabilistic prediction generated for a time step provides a respective probability distribution for a respective future location of the dynamic object at the time step.
2. The object motion prediction system of claim 1 , wherein the prediction head of the machine-learned object motion prediction model comprises a discrete residual prediction head that autoregressively generates the plurality of probabilistic predictions such that a particular probabilistic prediction is conditioned on a previous probabilistic prediction.
3. The object motion prediction system of claim 2 , wherein the discrete residual prediction head comprises a discrete residual flow head that, for the time step, takes the feature map and a previous potential from a previous time step as input and predicts a residual update to the previous potential from the previous time step.
4. The object motion prediction system of claim 2 , wherein the discrete residual prediction head comprises a discrete residual refinement head that, for the time step, takes a log normalized version of the feature map and log normalized version of a previous potential from a previous time step as input and predicts a refinement update to an initialized current potential for the time step.
5. The object motion prediction system of claim 1 , wherein the backbone portion of the machine-learned object motion prediction model comprises a feature pyramid network.
6. The object motion prediction system of claim 1 , wherein the scene rasterization encodes historical information about historical locations of the dynamic object within the scene.
7. The object motion prediction system of claim 1 , wherein the scene rasterization encodes semantic information about locations within the scene.
8. The object motion prediction system of claim 1 , wherein the scene rasterization comprises a plurality of discrete portions, and wherein the respective probability distribution for the time step comprises discrete probabilities respectively for the plurality of discrete portions of the scene rasterization.
9. The object motion prediction system of claim 1 , wherein the operations further comprise training the machine-learned object motion prediction model on training data, wherein training the machine-learned object motion prediction model comprises iteratively modifying parameter values of the machine-learned object motion prediction model to minimize a negative log likelihood of observed sequences of object movement described by the training data.
10. An autonomous vehicle, comprising:
at least one sensor configured to capture sensor data descriptive of a scene surrounding the autonomous vehicle, wherein the scene comprises a dynamic object; and
a computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that store:
a machine-learned object motion prediction model configured to generate probabilistic predictions for object motion over a plurality of time steps; and
instructions that are executable by the one or more processors to cause the computing system to perform operations, the operations comprising:
processing the sensor data to identify one or more observed locations of the dynamic object within the scene;
generating a scene rasterization that describes the scene that includes the dynamic object, wherein the scene rasterization encodes the one or more observed locations of the dynamic object;
processing the scene rasterization with the machine-learned object motion prediction model to generate a plurality of probabilistic predictions respectively for the plurality of time steps, wherein the probabilistic prediction generated for a time step provides a respective probability distribution for a respective future location of the dynamic object at the time step;
generating a motion plan for the autonomous vehicle based at least in part on the plurality of probabilistic predictions, wherein generating the motion plan for the autonomous vehicle comprises evaluating one or more cost functions using the plurality of probabilistic predictions; and
controlling motion of the autonomous vehicle based at least in part on the motion plan.
11. The autonomous vehicle of claim 10 , wherein the machine-learned object motion prediction model comprises a discrete residual prediction head that autoregressively generates the plurality of probabilistic predictions such that a probabilistic prediction is conditioned on a previous probabilistic prediction.
12. The autonomous vehicle of claim 10 , wherein the scene rasterization encodes semantic information about locations within the scene.
13. The autonomous vehicle of claim 10 , wherein the respective probability distribution for the time step comprises discrete probabilities respectively for a plurality of discrete portions of the scene rasterization.
14. The autonomous vehicle of claim 10 , wherein the machine-learned object motion prediction model has been trained on training data to minimize a negative log likelihood of observed sequences of object movement described by the training data.
15. A computer-implemented method for object motion prediction, the method comprising:
obtaining data that describes a scene surrounding an autonomous vehicle, wherein the scene includes a dynamic object;
generating a scene rasterization that encodes one or more historically observed locations of the dynamic object within the scene; and
processing the scene rasterization with a machine-learned object motion prediction model to generate a plurality of probabilistic predictions respectively for a plurality of time steps, wherein the probabilistic prediction generated for a time step provides a respective probability distribution for a respective future location of the dynamic object at the time step.
16. The computer-implemented method of claim 15 , wherein the scene rasterization comprises a two-dimensional bird's eye view of the scene.
17. The computer-implemented method of claim 15 , wherein the scene rasterization encodes semantic information about the scene.
18. The computer-implemented method of claim 15 , wherein the machine-learned object motion prediction model comprises a discrete residual prediction head that autoregressively generates the plurality of probabilistic predictions such that a probabilistic prediction is conditioned on a previous probabilistic prediction.
19. The computer-implemented method of claim 18 , wherein the discrete residual prediction head comprises a discrete residual flow head that takes the feature map and a previous potential from the previous time step as input.
20. The computer-implemented method of claim 18 , wherein the discrete residual prediction head comprises a discrete residual refinement head that takes a log normalized version of the feature map and log normalized version of a previous potential from the previous time step as input.