Sensor fusion for object detection
Techniques for fusing sensor data generated by different sensor modalities to improve object detections and object predictions determined by low-level systems of a vehicle. The techniques may include determining feature maps based on sensor data generated by different sensor modalities associated with a vehicle. In some examples, the feature maps may include at least a first feature map indicative of a location of an object in an environment of the vehicle and a second feature map indicative of elevation information associated with the object. The techniques may also include inputting the first feature map and the second feature map into a machine-learned model associated with the low-level system of the vehicle. In some examples, an output may be received from the machine-learned model that includes an occupancy grid, and the occupancy grid may exclude representation(s) associated with over-drivable object(s) and/or under-drivable object(s) that may be disposed in the environment.
1. A vehicle comprising:
a primary computing system comprising first hardware directed to a first task; and
a secondary computing system, the secondary computing system comprising second hardware directed to a second task, the secondary computing system configured to perform operations comprising:
receiving sensor data generated by different sensor modalities of the vehicle, the sensor data including at least radar data generated by a radar sensor and lidar data generated by a lidar sensor;
inputting the sensor data into a machine-learned model;
receiving, from the machine-learned model, an occupancy grid associated with an environment in proximity to the vehicle;
determining a corrective action for the vehicle based on the occupancy grid, wherein the corrective action is different than an action determined for the vehicle based on an output from the primary computing system and the corrective action comprises altering a planned trajectory of the vehicle, the planned trajectory determined by a planner component of the vehicle based on the output from the primary computing system of the vehicle; and
controlling, based at least in part on the corrective action, the vehicle.
2. The vehicle of claim 1 , the operations further comprising:
determining a radar feature map based on the radar data, wherein radar features included in the radar feature map are indicative of locations of objects in the environment surrounding the vehicle; and
determining a lidar feature map based on the lidar data, wherein lidar features included in the lidar feature map are indicative of elevation measurements associated with at least one of the radar features or the objects;
wherein inputting the sensor data into the machine-learned model comprises inputting the radar feature map and the lidar feature map into the machine-learned model.
3. The vehicle of claim 1 , wherein the occupancy grid includes at least a first representation of a non-drivable object in the environment and excludes a second representation of at least one of an over-drivable object or an under-drivable object in the environment.
4. The vehicle of claim 1 , wherein the occupancy grid is a first occupancy grid of a series of occupancy grids, the first occupancy grid corresponding with a present time and a second occupancy grid of the series of occupancy grids corresponding with a future time.
5. A method comprising:
inputting, by a secondary computing system comprising second hardware directed to a second task associated with a vehicle, radar data and lidar data into a machine-learned model;
receiving, from the machine-learned model, an occupancy grid associated with an environment in proximity to the vehicle;
determining, by the secondary computing system and based at least in part on the occupancy grid, whether to perform a corrective action for the vehicle, wherein the corrective action comprises altering a planned trajectory of the vehicle, the planned trajectory determined by a planner component associated with the vehicle based on an output from a primary computing system comprising first hardware directed to a first task associated with the vehicle; and
controlling, based at least in part on the corrective action, the vehicle.
6. The method of claim 5 , wherein the corrective action is different than an action determined for the vehicle based on the output from the primary computing system of the vehicle.
7. The method of claim 5 , further comprising:
determining, by the secondary computing system, a radar feature map based on the radar data, wherein radar features included in the radar feature map are indicative of locations of objects in the environment surrounding the vehicle; and
determining, by the secondary computing system, a lidar feature map based on the lidar data, wherein lidar features included in the lidar feature map are indicative of elevation measurements associated with at least one of the radar features or the objects;
wherein inputting the radar data and the lidar data into the machine-learned model comprises inputting the radar feature map and the lidar feature map into the machine-learned model.
8. The method of claim 7 , wherein:
the radar feature map includes a first point cloud of features corresponding with an object at the location, and
the lidar feature map includes a second point cloud of features that are indicative of elevation information corresponding with the first point cloud of features.
9. The method of claim 5 , wherein the machine-learned model is configured to:
determine a location of an object in the environment based at least in part on the radar data;
determine an elevation of the object at the location based at least in part on the lidar data; and
determine whether the object is at least one of an over-drivable object, an under-drivable object, or a non-drivable object based at least in part on the location and the elevation of the object.
10. The method of claim 5 , wherein the occupancy grid is a first occupancy grid of a series of occupancy grids output by the machine-learned model, the first occupancy grid corresponding with a present time and a second occupancy grid of the series of occupancy grids corresponding with a future time.
11. The method of claim 5 , wherein inputting the radar data and lidar data into the machine-learned model comprises inputting a series of frames of radar data and lidar data into the machine-learned model, each frame of the series of frames being associated with a different point in time.
12. One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors associated with a secondary computing system comprising second hardware directed to a second task of a vehicle to perform operations comprising:
inputting radar data and lidar data into a machine-learned model;
receiving, from the machine-learned model, an occupancy grid associated with an environment in proximity to the vehicle;
determining, based at least in part on the occupancy grid, whether to perform a corrective action for the vehicle, wherein the corrective action comprises altering a planned trajectory of the vehicle, the planned trajectory determined by a planner component associated with the vehicle based on an output from a primary computing system comprising first hardware directed to a first task associated with the vehicle; and
controlling, based at least in part on the corrective action, the vehicle.
13. The one or more non-transitory computer-readable media of claim 12 , wherein the corrective action is different than an action determined for the vehicle based on the output from the primary computing system of the vehicle.
14. The one or more non-transitory computer-readable media of claim 12 , the operations further comprising:
determining a radar feature map based on the radar data, wherein radar features included in the radar feature map are indicative of locations of objects in the environment surrounding the vehicle; and
determining a lidar feature map based on the lidar data, wherein lidar features included in the lidar feature map are indicative of elevation measurements associated with at least one of the radar features or the objects;
wherein inputting the radar data and the lidar data into the machine-learned model comprises inputting the radar feature map and the lidar feature map into the machine-learned model.
15. The one or more non-transitory computer-readable media of claim 14 , wherein:
the radar feature map includes a first point cloud of features corresponding with the object at the location, and
the lidar feature map includes a second point cloud of features that are indicative of the elevation measurements corresponding with the first point cloud of features.
16. The one or more non-transitory computer-readable media of claim 12 , wherein the machine-learned model is configured to:
determine a location of an object in the environment based at least in part on the radar data;
determine an elevation of the object at the location based at least in part on the lidar data; and
determine whether the object is at least one of an over-drivable object, an under-drivable object, or a non-drivable object based at least in part on the location and the elevation of the object.
17. The one or more non-transitory computer-readable media of claim 12 , wherein the occupancy grid is a first occupancy grid of a series of occupancy grids output by the machine-learned model, the first occupancy grid corresponding with a present time and a second occupancy grid of the series of occupancy grids corresponding with a future time.
18. The one or more non-transitory computer-readable media of claim 12 , wherein inputting the radar data and lidar data into the machine-learned model comprises inputting a series of frames of radar data and lidar data into the machine-learned model, each frame of the series of frames being associated with a different point in time.