Systems and methods for integrating radar data for improved object detection in autonomous vehicles

1. A computer-implemented method for generating perception and prediction data for autonomous vehicles, the method comprising:

accessing radar sensor data for an area around an autonomous vehicle, the radar sensor data including a plurality of radar points;

accessing LIDAR sensor data for the area around the autonomous vehicle;

generating, using one or more machine-learned models, a fused representation of the area around the autonomous vehicle based on the radar sensor data and the LIDAR sensor data;

determining, using the one or more machine-learned models, one or more objects in the area of the autonomous vehicle based on the fused representation;

for a respective object in the one or more objects:

determining a plurality of radar points from the radar sensor data that are associated with the respective object, each radar point having an associated velocity;

generating, using a machine-learned model, a similarity score for each determined radar point;

generating a weight associated with each radar point based, at least in part, on the similarity score; and

calculating a predicted velocity for the respective object based on a weighted average of the velocities of the plurality of radar points determined to be associated with the respective object;

generating a proposed motion plan based on the predicted velocity for the respective object; and

transmitting vehicle motion controls to one or more vehicle control systems to implement the motion plan.

2. The computer-implemented method of claim 1 , wherein the radar sensor data includes, for each radar point, a location of the radar point and a velocity associated with the radar point.

3. The computer-implemented method of claim 1 , wherein the radar sensor data includes data from a plurality of cycles of the radar sensor.

4. The computer-implemented method of claim 1 , wherein the LIDAR sensor data includes a plurality of LIDAR points, each LIDAR point having an associated location.

5. The computer-implemented method of claim 1 , wherein the LIDAR sensor data includes data from a plurality of sweeps of the LIDAR sensor.

6. The computer-implemented method of claim 1 , wherein generating, using one or more machine-learned models, a fused representation of the area around the autonomous vehicle based on the radar sensor data and the LIDAR sensor data further comprises:

generating a voxel grid representation of the radar sensor data; and

generating a voxel grid representation of the LIDAR sensor data.

7. The computer-implemented method of claim 6 , wherein the voxel grid representation of the radar sensor data includes a plurality of voxels and each voxel is associated with a voxel occupancy value.

8. The computer-implemented method claim 7 , wherein the voxel occupancy value for a respective voxel is based on a number of radar points that fall within the area associated with the respective voxels.

9. The computer-implemented method of claim 6 , wherein generating, using one or more machine-learned models, a fused representation of the area around the autonomous vehicle based on the radar sensor data and the LIDAR sensor data further comprises:

identifying, using the voxel grid representation of the radar sensor data as input to one or more machine-learned models, feature data for the radar sensor data; and

identifying, using the voxel grid representation of the LIDAR sensor data as input to one or more machine-learned models, feature data for the LIDAR sensor data.

10. The computer-implemented method of claim 1 , wherein generating, using one or more machine-learned models, a fused representation of the area around the autonomous vehicle based on the radar sensor data and the LIDAR sensor data further comprises:

concatenating feature data associated with the radar sensor data and feature data associated with the LIDAR sensor data and associating the concatenated data with a corresponding point in the fused representation.

11. The computer-implemented method of claim 1 , further comprising:

filtering the radar sensor data to remove radar points that have an associated velocity value below a predetermined threshold.

12. A computing system for generating perception and prediction data for autonomous vehicles, the system comprising:

one or more processors and one or more non-transitory computer-readable memories;

wherein the one or more non-transitory computer-readable memories store instructions that, when executed by the processor, cause the computing system to perform operations, the operations comprising:

accessing radar sensor data for an area around an autonomous vehicle, the radar sensor data including a plurality of radar points;

accessing LIDAR sensor data for the area around the autonomous vehicle;

generating, using one or more machine-learned models, a fused representation of the area around the autonomous vehicle based on the radar sensor data and the LIDAR data;

determining, using the one or more machine-learned models, one or more objects in the area of the autonomous vehicle based on the fused representation;

for a respective object in the one or more objects:

determining a plurality of radar points from the radar sensor data that are associated with the respective object, each radar point having an associated velocity;

generating, using a machine-learned model, a similarity score for each determined radar point;

generating a weight associated with each radar point based on the similarity score; and

calculating a predicted velocity for the respective object based on a weighted average of the velocities of the plurality of radar points determined to be associated with the respective object;

generating a proposed motion plan based on the predicted velocity for the respective object; and

transmitting vehicle motion controls to one or more vehicle control systems to implement the motion plan.

13. The computing system of claim 12 , wherein determining a plurality of radar points from the radar sensor data that are associated with the respective object, each radar point having an associated velocity further comprises:

determining for each radar point in the plurality of radar points, a distance from the radar point to a point associated with the respective object; and

selecting one or more radar points based on the determined distance between each radar point and a point associated with the respective object.

14. The computing system of claim 13 , wherein the point associated with the respective object is an estimated center of the object.

15. The computing system of claim 12 , wherein the respective object has an estimated direction.

16. The computing system of claim 15 , wherein generating a similarity score for each determined radar point further comprises:

generating for a radar point, a modified velocity value based on the estimated direction of the respective object and the velocity associated with the radar point.

17. The computing system of claim 16 , wherein the modified velocity value is parallel to the estimated direction of the respective object.

18. The computing system of claim 12 , wherein generating a weight associated with each radar point based on the similarity score further comprises:

normalizing the weights associated with the plurality of radar points.

19. An autonomous vehicle, comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:

accessing radar sensor data for an area around an autonomous vehicle, the radar sensor data including a plurality of radar points;

accessing LIDAR sensor data for the area around the autonomous vehicle;

generating, using one or more machine-learned models, a fused representation of the area around the autonomous vehicle based on the radar sensor data and the LIDAR data;

determining, using the one or more machine-learned models, one or more objects in the area of the autonomous vehicle based on the fused representation;

for a respective object in the one or more objects:

determining a plurality of radar points from the radar sensor data that are associated with the respective object, each radar point having an associated velocity;

generating, using a machine-learned model, a similarity score for each determined radar point;

generating a weight associated with each radar point based on the similarity score; and

calculating a predicted velocity for the respective object based on a weighted average of the velocities of the plurality of radar points determined to be associated with the respective object; and

generating a proposed motion plan based on the predicted velocity for the respective object; and

transmitting vehicle motion controls to one or more vehicle control systems to implement the motion plan.