Multi-task multi-sensor fusion for three-dimensional object detection
Provided are systems and methods that perform multi-task and/or multi-sensor fusion for three-dimensional object detection in furtherance of, for example, autonomous vehicle perception and control. In particular, according to one aspect of the present disclosure, example systems and methods described herein exploit simultaneous training of a machine-learned model ensemble relative to multiple related tasks to learn to perform more accurate multi-sensor 3D object detection. For example, the present disclosure provides an end-to-end learnable architecture with multiple machine-learned models that interoperate to reason about 2D and/or 3D object detection as well as one or more auxiliary tasks. According to another aspect of the present disclosure, example systems and methods described herein can perform multi-sensor fusion (e.g., fusing features derived from image data, light detection and ranging (LIDAR) data, and/or other sensor modalities) at both the point-wise and region of interest (ROI)-wise level, resulting in fully fused feature representations.
1. A method for fusion of sensor data descriptive of an environment, the method comprising:
generating a LIDAR feature map using a machine-learned LIDAR processing model and based at least in part on a LIDAR point cloud descriptive of a region of interest associated with the environment;
generating an image feature map using a machine-learned image processing model and based at least in part on image data descriptive of the region of interest;
fusing the LIDAR feature map and the image feature map; and
generating, based on the fused LIDAR feature map and image feature map, a depth completion map descriptive of the region of interest;
wherein the machine-learned LIDAR processing model and the machine-learned image processing model were jointly trained end-to-end based on a loss function evaluated at least in part by comparing reference depth completion maps and depth completion maps generated using the machine-learned LIDAR processing model and the machine-learned image processing model;
wherein the machine-learned LIDAR processing model and the machine-learned image processing model were jointly trained end-to-end by:
backpropagating the loss function through the machine-learned LIDAR processing model; and
modifying, based on the backpropagation of the loss function, one or more weights associated with the machine-learned LIDAR processing model.
2. The method of claim 1 , wherein:
the LIDAR point cloud is represented in bird's-eye-view space;
the image data is represented in two-dimensional image space; and
the depth completion map is represented in bird's-eye-view space.
3. The method of claim 1 , wherein the depth completion map is generated using a machined-learned depth-completion model.
4. The method of claim 2 , wherein the depth completion map is generated by associating respective pixels of the fused LIDAR feature map and image feature map with respective locations in bird's-eye-view space.
5. The method of claim 2 , wherein the depth completion map contains depth information not contained in the LIDAR point cloud.
6. The method of claim 1 , wherein the region of interest corresponds to a subset of the sensor data.
7. The method of claim 1 , comprising:
determining a location for an object in the environment based at least in part on the depth completion map.
8. An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that are executable by the one or more processors to cause the autonomous vehicle control system to perform operations, the operations comprising:
generating a LIDAR feature map using a machine-learned LIDAR processing model and based at least in part on a LIDAR point cloud descriptive of a region of interest associated with an environment;
generating an image feature map using a machine-learned image processing model and based at least in part on image data descriptive of the region of interest;
fusing the LIDAR feature map and the image feature map; and
generating, based on the fused LIDAR feature map and image feature map, a depth completion map descriptive of the region of interest;
wherein the machine-learned LIDAR processing model and the machine-learned image processing model were jointly trained end-to-end based on a loss function evaluated at least in part by comparing reference depth completion maps and depth completion maps generated using the machine-learned LIDAR processing model and the machine-learned image processing model;
wherein the machine-learned LIDAR processing model and the machine-learned image processing model were jointly trained end-to-end by:
backpropagating the loss function through the machine-learned LIDAR processing model; and
modifying, based on the backpropagation of the loss function, one or more weights associated with the machine-learned LIDAR processing model.
9. The autonomous vehicle control system of claim 8 , wherein:
the LIDAR point cloud is represented in bird's-eye-view space;
the image data is represented in two-dimensional image space; and
the depth completion map is represented in bird's-eye-view space.
10. The autonomous vehicle control system of claim 8 , wherein the depth completion map is generated using a machined-learned depth-completion model.
11. The autonomous vehicle control system of claim 9 , wherein the depth completion map is generated by associating respective pixels of the fused LIDAR feature map and image feature map with respective locations in bird's-eye-view space.
12. The autonomous vehicle control system of claim 8 , wherein the depth completion map contains depth information not contained in the LIDAR point cloud.
13. The autonomous vehicle control system of claim 8 , wherein the region of interest corresponds to a subset of sensor data descriptive of the environment.
14. The autonomous vehicle control system of claim 8 , wherein the operations comprise:
determining a location for an object in the environment based at least in part on the depth completion map.
15. The autonomous vehicle control system of claim 14 , wherein the operations comprise:
generating a motion plan for the autonomous vehicle based at least in part on the location; and
controlling the autonomous vehicle according to the motion plan.
16. One or more non-transitory computer-readable media storing instructions that are executable by one or more processors to cause an autonomous vehicle control system to perform operations, the operations comprising:
generating a LIDAR feature map using a machine-learned LIDAR processing model and based at least in part on a LIDAR point cloud descriptive of a region of interest associated with an environment;
generating an image feature map using a machine-learned image processing model and based at least in part on image data descriptive of the region of interest;
fusing the LIDAR feature map and the image feature map; and
generating, based on the fused LIDAR feature map and image feature map, a depth completion map descriptive of the region of interest;
wherein the machine-learned LIDAR processing model and the machine-learned image processing model were jointly trained end-to-end based on a loss function evaluated at least in part by comparing reference depth completion maps and depth completion maps generated using the machine-learned LIDAR processing model and the machine-learned image processing model;
wherein the machine-learned LIDAR processing model and the machine-learned image processing model were jointly trained end-to-end by:
backpropagating the loss function through the machine-learned LIDAR processing model; and
modifying, based on the backpropagation of the loss function, one or more weights associated with the machine-learned LIDAR processing model.
17. The one or more non-transitory computer-readable media of claim 16 , wherein:
the LIDAR point cloud is represented in bird's-eye-view space;
the image data is represented in two-dimensional image space; and
the depth completion map is represented in bird's-eye-view space.
18. The one or more non-transitory computer-readable media of claim 17 , wherein the depth completion map is generated by associating respective pixels of the fused LIDAR feature map and image feature map with respective locations in bird's-eye-view space.
19. The one or more non-transitory computer-readable media of claim 16 , wherein the operations comprise:
determining a location for an object in the environment based at least in part on the depth completion map.
20. The method of claim 1 , wherein the machine-learned LIDAR processing model and the machine-learned image processing model were jointly trained end-to-end by:
computing the loss function to measure a difference between a reference depth completion map and a depth completion map generated using the fused LIDAR feature map and image feature map;
backpropagating the loss function through:
the machine-learned LIDAR processing model, and
the machine-learned image processing model; and
modifying, based on the backpropagation of the loss function:
one or more weights associated with the machine-learned LIDAR processing model; and
one or more weights associated with the machine-learned image processing model.