Systems and methods for latent distribution modeling for scene-consistent motion forecasting

1. A computer-implemented method for determining scene-consistent motion forecasts from sensor data, the method comprising:

obtaining, by a computing system comprising one or more computing devices, scene data comprising one or more actor features;

providing, by the computing system, the scene data to a latent prior model, the latent prior model configured to generate scene latent data in response to receipt of scene data, the scene latent data comprising a latent distribution that is partitioned into one or more latent variables, wherein the one or more latent variables encode unobserved dynamics relative to a respective actor feature of the one or more actor features;

obtaining, by the computing system, the scene latent data from the latent prior model;

sampling, by the computing system, latent sample data from the scene latent data;

providing, by the computing system, the latent sample data to a decoder model, the decoder model configured to decode the latent sample data into a motion forecast comprising one or more predicted trajectories of the one or more actor features; and

receiving, by the computing system, the motion forecast comprising one or more predicted trajectories of the one or more actor features from the decoder model;

providing, by the computing system, the one or more predicted trajectories to a motion planning model configured to generate a motion plan for an autonomous vehicle based at least in part on the one or more predicted trajectories; and

implementing, by the computing system, the motion plan to control the autonomous vehicle.

2. The computer-implemented method of claim 1 , wherein the decoder model comprises a deterministic decoder model.

3. The computer-implemented method of claim 1 , wherein the decoder model comprises a specified and tractable conditional likelihood.

4. The computer-implemented method of claim 1 , wherein the one or more latent variables comprise one or more continuous latent variables.

5. The computer-implemented method of claim 1 , further comprising:

obtaining, by the computing system, one or more scene observations;

providing, by the computing system, the one or more scene observations to a scene feature extraction model, the scene feature extraction model comprising one or more neural networks configured to extract one or more scene features from the one or more scene observations;

receiving, by the computing system, the one or more scene features from the scene feature extraction model;

providing, by the computing system, the one or more scene features to an actor feature recognition model, the actor feature recognition model configured to:

extract spatial feature maps for bounding boxes from the one or more scene features by rotated region of interest align;

pool a region around each spatial feature map to produce pooled actor features;

downsample the pooled actor features by applying one or more downsampling convolutional neural networks; and

max-pool along spatial dimensions to reduce each pooled actor feature to a respective actor feature of the one or more actor features; and

receiving, by the computing system, the one or more actor features from the actor feature recognition model.

6. The computer-implemented method of claim 1 , further comprising:

sampling, by the computing system, second latent sample data from the scene latent data;

providing, by the computing system, the second latent sample data to the decoder model; and

receiving, by the computing system, a second motion forecast comprising one or more second predicted trajectories of the one or more actor features from the decoder model.

7. The computer-implemented method of claim 1 , wherein at least one of the latent prior model or the decoder model comprises a scene interaction model configured to model the latent distribution as an interaction graph comprising one or more nodes representative of the one or more actor features and one or more edges representative of interactions between the one or more actor features.

8. The computer-implemented method of claim 7 , wherein the scene interaction model comprises one or more graph neural networks.

9. The computer-implemented method of claim 8 , wherein a message function of the one or more graph neural networks comprises a multi-layer perceptron model that takes as input one or more terminal nodes of the one or more nodes at a previous propagation step of the one or more graph neural networks.

10. The computer-implemented method of claim 8 , wherein an aggregation function of the one or more graph neural networks comprises a feature-wise max-pooling aggregation function.

11. The computer-implemented method of claim 8 , wherein a gated recurrent unit cell is configured to update a state of the one or more nodes.

12. The computer-implemented method of claim 1 , wherein the one or more actor features comprise data descriptive of a context of one or more traffic participants.

13. The computer-implemented method of claim 1 , wherein the unobserved dynamics relative to the respective actor feature of the one or more actor features comprise at least one of interactions with neighboring actor features of the respective actor feature or traffic rules that apply in a locality of the respective actor feature.

14. A computer-implemented method of training a motion forecasting system, the method comprising:

obtaining, by a computing system comprising one or more computing devices, a training dataset comprising one or more training examples labeled with ground truth data, the one or more training examples comprising one or more actor features and the ground truth data comprising a ground truth context of the one or more actor features;

providing, by the computing system, the one or more training examples labeled with ground truth data to a latent encoder model, the latent encoder model configured to produce a first latent distribution in response to receipt of the one or more training examples and the ground truth data;

providing, by the computing system, the one or more training examples to a latent prior model, the latent prior model configured to produce a second latent distribution in response to receipt of the one or more training examples;

determining, by the computing system, a training loss based at least in part on the first latent distribution and the second latent distribution; and

backpropagating, by the computing system, the training loss through at least the latent prior model to train at least the latent prior model.

15. The computer-implemented method of claim 14 , wherein the training loss comprises a KL divergence loss between the first latent distribution and the second latent distribution.

16. The computer-implemented method of claim 14 , wherein the method further comprises:

providing, by the computing system, training scene observations to a feature extraction model;

receiving, by the computing system, one or more predicted features from the feature extraction model; and

determining, by the computing system, a feature loss between the one or more predicted features and the ground truth data;

wherein the training loss comprises the feature loss.

17. The computer-implemented method of claim 16 , wherein the feature loss comprises a cross-entropy loss between the one or more predicted features and one or more training features of the ground truth data and a regression loss between bounding boxes of the one or more predicted features and bounding boxes of the ground truth data.

18. The computer-implemented method of claim 14 , further comprising:

sampling, by the computing system, the first latent distribution to produce one or more first latent samples;

sampling, by the computing system, the second latent distribution to produce one or more second latent samples;

providing, by the computing system, the one or more first latent samples to a decoder model;

receiving, by the computing system, one or more first predicted trajectories from the decoder model;

providing, by the computing system, the one or more second latent samples to the decoder model;

receiving, by the computing system, one or more second predicted trajectories from the decoder model; and

determining, by the computing system, a reconstruction loss based at least in part on the one or more first predicted trajectories and the one or more second predicted trajectories;

wherein the training loss comprises the reconstruction loss.

19. The computer-implemented method of claim 18 , wherein the reconstruction loss comprises a Huber loss.

20. A computer-implemented method of operating an autonomous vehicle, the method comprising:

obtaining, by the computing system, one or more scene observations;

providing, by the computing system, the one or more scene observations to a feature extraction model, the feature extraction model configured to produce scene data comprising one or more actor features from the one or more scene observations;

receiving, by the computing system, the scene data comprising one or more actor features from the feature extraction model;

providing, by the computing system, the scene data to a latent prior model, the latent prior model configured to generate scene latent data in response to receipt of scene data, the scene latent data comprising a latent distribution that is partitioned into one or more latent variables, wherein the one or more latent variables encode unobserved dynamics relative to a respective actor feature of the one or more actor features;

obtaining, by the computing system, the scene latent data from the latent prior model;

sampling, by the computing system, one or more latent samples from the scene latent data;

providing, by the computing system, the one or more latent samples to a decoder model, the decoder model configured to decode the latent samples into a motion forecast comprising one or more predicted trajectories of the one or more actor features;

obtaining, by the computing system, one or more motion forecasts comprising one or more predicted trajectories of the one or more actor features from the decoder model;

providing, by the computing system, the one or more predicted trajectories to a motion planning model configured to generate a motion plan for an autonomous vehicle based at least in part on the one or more predicted trajectories; and

implementing, by the computing system, the motion plan to control the autonomous vehicle.