Systems and methods for end-to-end trajectory prediction using radar, LIDAR, and maps

1. A computing system, comprising:

one or more processors;

a memory comprising one or more tangible non-transitory computer-readable media, the media storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising:

obtaining, from one or more sensors of an autonomous vehicle, LIDAR data and radar data, the radar data comprising one or more radar sweeps, the LIDAR data comprising one or more LIDAR sweeps;

inputting the LIDAR data and the radar data into a network model comprising one or more machine-leaned models;

generating, by the network model, one or more features for each of the LIDAR data and the radar data, wherein generating the one or more features for the radar data comprises determining a feature vector for each cell in a coordinate frame and concatenating per sweep feature vectors for each cell;

combining, by the network model, the one or more generated features for each of the LIDAR data and the radar data to generate fused feature data;

generating, by the network model, prediction data based at least in part on the fused feature data; and

receiving, as an output of the network model, the prediction data, the prediction data comprising a respective predicted trajectory for a future time period for one or more detected objects.

2. The computing system of claim 1 , wherein the operations further comprise:

determining a motion plan based at least in part on the respective predicted trajectory for the one or more detected objects; and controlling an autonomous vehicle based at least in part on the motion plan.

3. The computing system of claim 1 , wherein combining, by the network model, the one or more generated features for each of the LIDAR data and the radar data to generate the fused feature data comprises:

receiving, as an output of a respective subnetwork of the network model, one or more respective generated features for each of the LIDAR data and the radar data;

inputting the one or more respective generated features for the LIDAR data and the radar data into a multi-scale fusion backbone of the network model; and

concatenating, by the multi-scale fusion backbone, the one or more respective generated features for the LIDAR data and the radar data to generate the fused feature data.

4. The computing system of claim 1 , wherein the operations further comprise:

transforming the radar data into the coordinate frame associated with a most recent radar sweep in the radar data, the coordinate frame comprising a plurality of cells.

5. The computing system of claim 4 , wherein the coordinate frame comprises a two-dimensional top-down grid comprising a cartesian grid or an arc grid.

6. The computing system of claim 1 , wherein the feature vector for each cell in the coordinate frame for each sweep in the radar data is further determined using a graph between the cell and the radar data using a nearest neighbor algorithm.

7. The computing system of claim 1 , wherein generating, by the network model, the one or more features for the LIDAR data comprises:

discretizing LIDAR data points into an occupancy coordinate frame comprising a plurality of cells for each sweep; and

concatenating per sweep occupancy features for each cell.

8. The computing system of claim 1 , wherein the operations further comprise:

obtaining map data associated with the LIDAR data and the radar data; and

generating, by the network model, one or more features for the map data comprises rasterizing the map data into a coordinate frame comprising a plurality of cells; and

wherein the fused feature data is generated by combining the one or more features for the map data with the one or more features for each of the LIDAR data and the radar data.

9. The computing system of claim 1 , wherein the network model has been trained form end-to-end using a loss function comprising a detection loss parameter and a trajectory loss parameter.

10. The computing system of claim 9 , wherein the detection loss parameter comprises a classification loss parameter and a regression loss parameter.

11. The computing system of claim 1 , wherein the prediction data comprises a probabilistic multi-hypothesis trajectory prediction for each of the one or more detected objects.

12. The computing system of claim 1 , wherein the prediction data comprises a single hypothesis trajectory prediction for each of the one or more detected objects.

13. The computing system of claim 1 , wherein generating, by the network model, the prediction data based at least in part on the fused feature data comprises inputting the fused feature data into a prediction head of the network model.

14. A computer-implemented method for training a trajectory prediction network model from end-to-end, the network model comprising a plurality of subnetworks configured to extract one or more features from input data and a trajectory prediction machine-learned model, the method comprising:

inputting training data into the network model, the training data comprising radar data comprising a plurality of sweeps, LIDAR data comprising a plurality of sweeps, and map data;

generating, by the network model, one or more features for each of the LIDAR data; the radar data, and the map data, wherein generating the one or more features for the radar data comprises determining a feature vector for each cell in a coordinate frame and concatenating per sweep feature vectors for each cell;

combining, by the network model, the one or more generated features for each of the LIDAR data, the radar data; and the map data to generate fused feature data;

predicting, by the network model, a respective future trajectory for one or more objects detected by the network model;

determining a loss function for the network model based at least in part on a comparison between the respective predicted future trajectory for the one or more Objects and a ground truth future trajectory; and training the network model based at least in part on the loss function.

15. The computer-implemented method for training the trajectory prediction network model from end-to-end of claim 14 , wherein the loss function comprises a detection loss parameter and a trajectory loss parameter.

16. The computer-implemented method for training the trajectory prediction network model from end-to-end of claim 15 , wherein the detection loss parameter comprises a classification loss parameter and a regression loss parameter.

17. The computer-implemented method for training the trajectory prediction network model from end-to-end of claim 14 , wherein the training data comprises LIDAR data and radar data acquired during, one or more autonomous vehicle driving sessions.

18. An autonomous vehicle, comprising:

one or more LIDAR sensors configured to obtain sweeps of LIDAR data;

one or more radar sensors configured to obtain sweeps of radar data; and

a computing system, comprising:

one or more processors;

a memory comprising one or more tangible non-transitory computer-readable media, the media storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising:

obtaining, from the one or more LIDAR sensors, LIDAR data comprising one or more sweeps;

obtaining, from the one or more radar sensors, radar data comprising one OF more sweeps;

obtaining map data associated with the LIDAR data and the radar data;

inputting the LIDAR data, the radar data, and the map data into a network model comprising one or more machine-learned models;

generating, by the network model, one or more features for each of the LIDAR data, the radar data, and the map data, wherein generating the one or more features for the radar data comprises determining a feature vector for each cell in a coordinate frame and concatenating per sweep feature vectors for each cell;

combining, by the network model, the one or more generated features for each of the LIDAR data, the radar data, and the map data to generate fused feature data;

generating, by the network model, prediction data based at least in part on the fused feature data;

receiving, as an output of the network model, the prediction data, the prediction data comprising a respective predicted trajectory for a future time period for one or more detected objects;

determining a motion plan based at least in part on the respective predicted trajectory for the one or more detected objects; and

controlling the autonomous vehicle based at least in part on the motion plan.

19. The autonomous vehicle of claim 18 , wherein generating, by the network model, the one or more features for the radar data comprises:

transforming the radar data into a coordinate frame associated with a most recent radar sweep in the radar data, the coordinate frame comprising a plurality of cells;

determining a feature vector for each cell in the coordinate frame for each sweep in the radar data; and

concatenating the per sweep feature vectors for each cell.