Multi-Task Multi-Sensor Fusion for Three-Dimensional Object Detection
Provided are systems and methods that perform multi-task and/or multi-sensor fusion for three-dimensional object detection in furtherance of, for example, autonomous vehicle perception and control. In particular, according to one aspect of the present disclosure, example systems and methods described herein exploit simultaneous training of a machine-learned model ensemble relative to multiple related tasks to learn to perform more accurate multi-sensor 3D object detection. For example, the present disclosure provides an end-to-end learnable architecture with multiple machine-learned models that interoperate to reason about 2D and/or 3D object detection as well as one or more auxiliary tasks. According to another aspect of the present disclosure, example systems and methods described herein can perform multi-sensor fusion (e.g., fusing features derived from image data, light detection and ranging (LIDAR) data, and/or other sensor modalities) at both the point-wise and region of interest (ROI)-wise level, resulting in fully fused feature representations.
1 . A computer-implemented method for detecting objects an environment, the method comprising:
generating LIDAR feature data using a machine-learned LIDAR processing model that processes an input LIDAR point cloud;
generating image feature data using a machine-learned image processing model that processes input image data;
fusing at least a portion of the LIDAR feature data and at least a portion of the image feature data to obtain fused feature data;
generating, using the fused feature data, an initial object detection estimate;
obtaining, using the initial object detection estimate, feature data describing a region of interest associated with the initial object detection estimate, wherein the feature data is selected from one or more of: the LIDAR feature data, the image feature data, and the fused feature data; and
generating a refined object detection using a machine-learned refinement model that processes the feature data describing the region of interest.
2 . The computer-implemented method of claim 1 , wherein fusing at least the portion of the LIDAR feature data and at least the portion of the image feature data comprises:
performing point-wise feature fusion to fuse at least the portion of the LIDAR feature data and at least the portion of the image feature data.
3 . The computer-implemented method of claim 1 , wherein the machine-learned refinement model comprises one or more fully connected layers.
4 . The computer-implemented method of claim 1 , wherein the initial object detection estimate comprises an initial bounding box for an object, and wherein the refined object detection comprises a refined bounding box for the object.
5 . The computer-implemented method of claim 1 , wherein the feature data describing the region of interest is obtained from an output layer of the machine-learned LIDAR processing model or an output layer of the machine-learned image processing model.
6 . The computer-implemented method of claim 1 , wherein generating the refined object detection comprises:
combining at least a portion of the image feature data and at least a portion of the LIDAR feature data; and
processing the combined portions of the image feature data and the LIDAR feature data using the machine-learned refinement model to generate the refined object detection.
7 . The computer-implemented method of claim 1 , wherein generating the refined object detection comprises:
regressing, using the machine-learned refinement model, a relative position refinement.
8 . The computer-implemented method of claim 1 , wherein generating the refined object detection comprises:
generating, using a first output layer of the machine-learned refinement model, a refined two-dimensional bounding box associated with the object; and
generating, using a second output layer of the machine-learned refinement model, a refined three-dimensional bounding box associated with the object.
9 . The computer-implemented method of claim 1 , wherein the machine-learned LIDAR processing model, the machine-learned image processing model, and the machine-learned refinement model were jointly trained end-to-end based on a total loss function that evaluates training object detections output by the machine-learned refinement model.
10 . The computer-implemented method of claim 1 , wherein the machine-learned image processing model comprises one or more pre-trained image processing layers.
11 . One or more non-transitory computer-readable media that store instructions that are executable by one or more processors to cause an autonomous vehicle control system to perform operations, wherein the operations comprise:
generating LIDAR feature data using a machine-learned LIDAR processing model that processes an input LIDAR point cloud;
generating image feature data using a machine-learned image processing model that processes input image data;
fusing at least a portion of the LIDAR feature data and at least a portion of the image feature data to obtain fused feature data;
generating, using the fused feature data, an initial object detection estimate;
obtaining, using the initial object detection estimate, feature data describing a region of interest associated with the initial object detection estimate, wherein the feature data is selected from one or more of: the LIDAR feature data, the image feature data, and the fused feature data; and
generating a refined object detection using a machine-learned refinement model that processes the feature data describing the region of interest.
12 . The one or more non-transitory computer-readable media of claim 11 , wherein fusing at least the portion of the LIDAR feature data and at least the portion of the image feature data comprises:
performing point-wise feature fusion to fuse at least the portion of the LIDAR feature data and at least the portion of the image feature data.
13 . The one or more non-transitory computer-readable media of claim 11 , wherein the machine-learned refinement model comprises one or more fully connected layers.
14 . The one or more non-transitory computer-readable media of claim 11 , wherein the initial object detection estimate comprises an initial bounding box for an object, and wherein the refined object detection comprises a refined bounding box for the object.
15 . The one or more non-transitory computer-readable media of claim 11 , wherein the feature data describing the region of interest is obtained from an output layer of the machine-learned LIDAR processing model or an output layer of the machine-learned image processing model.
16 . The one or more non-transitory computer-readable media of claim 11 , wherein generating the refined object detection comprises:
combining at least a portion of the image feature data and at least a portion of the LIDAR feature data; and
processing the combined portions of the image feature data and the LIDAR feature data using the machine-learned refinement model to generate the refined object detection.
17 . The one or more non-transitory computer-readable media of claim 11 , wherein generating the refined object detection comprises:
regressing, using the machine-learned refinement model, a relative position refinement.
18 . The one or more non-transitory computer-readable media of claim 11 , wherein generating the refined object detection comprises:
generating, using a first output layer of the machine-learned refinement model, a refined two-dimensional bounding box associated with the object; and
generating, using a second output layer of the machine-learned refinement model, a refined three-dimensional bounding box associated with the object.
19 . The one or more non-transitory computer-readable media of claim 11 , wherein the machine-learned LIDAR processing model, the machine-learned image processing model, and the machine-learned refinement model were jointly trained end-to-end based on a total loss function that evaluates training object detections output by the machine-learned refinement model.
20 . An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising:
one or more processors; and
one or more non-transitory computer-readable media that store instructions that are executable by the one or more processors to cause the autonomous vehicle control system to perform operations, wherein the operations comprise:
generating LIDAR feature data using a machine-learned LIDAR processing model that processes an input LIDAR point cloud;
generating image feature data using a machine-learned image processing model that processes input image data;
fusing at least a portion of the LIDAR feature data and at least a portion of the image feature data to obtain fused feature data;
generating, using the fused feature data, an initial object detection estimate;
obtaining, using the initial object detection estimate, feature data describing a region of interest associated with the initial object detection estimate, wherein the feature data is selected from one or more of: the LIDAR feature data, the image feature data, and the fused feature data; and
generating a refined object detection using a machine-learned refinement model that processes the feature data describing the region of interest.