Three-dimensional object detection
Generally, the disclosed systems and methods implement improved detection of objects in three-dimensional (3D) space. More particularly, an improved 3D object detection system can exploit continuous fusion of multiple sensors and/or integrated geographic prior map data to enhance effectiveness and robustness of object detection in applications such as autonomous driving. In some implementations, geographic prior data (e.g., geometric ground and/or semantic road features) can be exploited to enhance three-dimensional object detection for autonomous vehicle applications. In some implementations, object detection systems and methods can be improved based on dynamic utilization of multiple sensor modalities. More particularly, an improved 3D object detection system can exploit both LIDAR systems and cameras to perform very accurate localization of objects within three-dimensional space relative to an autonomous vehicle. For example, multi-sensor fusion can be implemented via continuous convolutions to fuse image data samples and LIDAR feature maps at different levels of resolution.
1. An autonomous vehicle (AV) control system, comprising:
one or more processors;
one or more non-transitory computer-readable media that store instructions for execution by the one or more processors that cause the AV control system to perform operations, the operations comprising:
(i) fusing a three-dimensional (3D) representation of light detection and ranging (LIDAR) data associated with an environment of an autonomous vehicle with a 3D representation of geographic map data to create a 3D representation of map-modified LIDAR data;
(ii) fusing a two-dimensional (2D) representation of image data associated with the environment of the autonomous vehicle with the 3D representation of map-modified LIDAR data to create a 3D feature map comprising image features from the 2D representation of the image data and map-modified LIDAR features from the 3D representation of map-modified LIDAR data, wherein (ii) comprises:
processing the 2D representation of the image data and the 3D representation of the map-modified LIDAR data by a machine-learned neural network that comprises one or more fusion layers; and
receiving, as an output of the machine-learned neural network, the 3D feature map; and
(iii) determining one or more object detections in the environment of the autonomous vehicle based on the 3D feature map.
2. The AV control system of claim 1 , wherein (ii) comprises transferring data points from pixels in the 2D representation of the image data into corresponding locations within the 3D representation of the map-modified LIDAR data.
3. The AV control system of claim 2 , wherein the corresponding locations within the 3D representation of the map-modified LIDAR data are determined from the pixels in the 2D representation of the image data using Euclidean distance.
4. The AV control system of claim 1 , wherein at least one of (i) or (ii) comprises implementing one or more continuous convolutions using a deep neural network.
5. The AV control system of claim 1 , wherein (i) comprises fusing radar data with the 3D representation of the LIDAR data and the 3D representation of the geographic map data.
6. The AV control system of claim 1 ,
wherein at least one of the one or more fusion layers is configured to fuse the image features at a first level of resolution with the map-modified LIDAR features at a second level of resolution that is different than the first level of resolution.
7. The AV control system of claim 1 , wherein the 2D representation of the image data is provided in a range view and the 3D representation of the map-modified LIDAR data is provided in a bird's eye view.
8. The AV control system of claim 1 , the operations further comprising:
rasterizing map data to generate the 3D representation of geographic map data.
9. The AV control system of claim 1 , the operations further comprising:
tracking the one or more object detections in the environment of the autonomous vehicle over time.
10. The AV control system of claim 1 , the operations further comprising:
determining a motion plan for the autonomous vehicle based on the one or more object detections; and
controlling a motion of the autonomous vehicle to execute the motion plan.
11. The AV control system of claim 1 , wherein the one or more object detections comprise a respective classification classifying the one or more object detections as corresponding to a particular class of object comprising at least one of: (i) a vehicle, (ii) a bicycle, (iii) a pedestrian, or (iv) another object.
12. The AV control system of claim 1 , wherein the one or more object detections respectively comprise a location and a confidence score associated with the location.
13. An autonomous vehicle, comprising:
a camera system configured to capture image data associated with an environment of an autonomous vehicle;
a RADAR system configured to capture radar data associated with the environment of the autonomous vehicle;
a LIDAR system configured to capture LIDAR point cloud data representing a three-dimensional (3D) view of the environment of the autonomous vehicle;
a map system configured to provide geographic data associated with the environment of the autonomous vehicle;
a fusion system configured to: (i) fuse the LIDAR point cloud data with the radar data and the geographic data into a 3D representation of map-modified sensor data; and (ii) fuse image features from the image data with map-modified sensor features from the map-modified sensor data to generate a 3D feature map comprising the fused image features and map-modified sensor features, wherein the fusion system is configured for (ii) to: provide the image data and the 3D representation of the map-modified sensor data as input to a machine-learned neural network that comprises one or more fusion layers, and receive, as an output of the machine-learned neural network, the 3D feature map;
an object detector configured to determine one or more object detections based on the 3D feature map.
14. The autonomous vehicle of claim 13 , wherein the fusion system comprises one or more deep neural networks configured to execute one or more continuous convolutions to fuse the image features from the image data with the map-modified sensor features from the map-modified sensor data.
15. The autonomous vehicle of claim 13 , wherein the fusion system is configured to transfer data points from pixels in the image data into corresponding locations within the 3D representation of the map-modified sensor data, the corresponding locations determined using Euclidean distance.
16. The autonomous vehicle of claim 13 ,
wherein at least one of the one or more fusion layers is configured to fuse the image features at a first level of resolution with the map-modified sensor features at a second level of resolution that is different than the first level of resolution.
17. The autonomous vehicle of claim 13 , wherein the one or more object detections respectively comprise: (i) a classification corresponding to a particular class of object, (ii) a location, and (iii) a confidence score associated with the location.
18. The autonomous vehicle of claim 13 , further comprising a control system configured to determine a motion plan for the autonomous vehicle based on the one or more object detections and control a motion of the autonomous vehicle to execute the motion plan.
19. A computer-implemented method, comprising:
(i) fusing a three-dimensional (3D) representation of light detection and ranging (LIDAR) data associated with an environment of an autonomous vehicle with a 3D representation of geographic map data to create a 3D representation of map-modified LIDAR data;
(ii) fusing a two-dimensional (2D) representation of image data associated with the environment of the autonomous vehicle with the 3D representation of map-modified LIDAR data to create a 3D feature map comprising image features from the 2D representation of the image data and map-modified LIDAR features from the 3D representation of map-modified LIDAR data, wherein (ii) comprises:
processing the 2D representation of the image data and the 3D representation of the map-modified LIDAR data by a machine-learned neural network that comprises one or more fusion layers; and
receiving, as an output of the machine-learned neural network, the 3D feature map; and
(iii) determining one or more object detections in the environment of the autonomous vehicle based on the 3D feature map.
20. The computer-implemented method of claim 19 , wherein (ii) comprises transferring data points from pixels in the 2D representation of image data into corresponding locations within the 3D representation of the map-modified LIDAR data, the corresponding locations determined using Euclidean distance.