Continuous convolution and fusion in neural networks
Systems and methods are provided for machine-learned models including convolutional neural networks that generate predictions using continuous convolution techniques. For example, the systems and methods of the present disclosure can be included in or otherwise leveraged by an autonomous vehicle. In one example, a computing system can perform, with a machine-learned convolutional neural network, one or more convolutions over input data using a continuous filter relative to a support domain associated with the input data, and receive a prediction from the machine-learned convolutional neural network. A machine-learned convolutional neural network in some examples includes at least one continuous convolution layer configured to perform convolutions over input data with a parametric continuous kernel.
1. An autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprising:
one or more processors; and
one or more non-transitory computer-readable media that store instructions for execution by the one or more processors to cause the one or more processors to perform operations, the operations comprising:
providing first sensor data and second sensor data of an environment of the autonomous vehicle to a neural network, wherein the first sensor data is obtained from a first sensor having a first modality, wherein the second sensor data is obtained from a second sensor having a second modality that is different than the first modality, and wherein the neural network comprises one or more continuous fusion layers configured to generate a dense representation of sensor data under a common modality;
processing the first sensor data and the second sensor data with the one or more continuous fusion layers to fuse information from a source domain associated with the first sensor into a target domain associated with the second sensor;
determining a motion plan for controlling a motion of the autonomous vehicle, wherein the motion plan is based at least in part on the fused information; and
controlling the motion of the autonomous vehicle based at least in part on the motion plan.
2. The autonomous vehicle control system of claim 1 , wherein the dense representation of data points is extracted from both the first modality of the first sensor and the second modality of the second sensor.
3. The autonomous vehicle control system of claim 1 , wherein processing the first sensor data and the second sensor data with the one or more continuous fusion layers is further configured to extract a first portion of data points in the source domain associated with the first sensor based on the target domain associated with the second sensor.
4. The autonomous vehicle control system of claim 3 , wherein the one or more continuous fusion layers use a nearest neighbor technique to extract the first portion of data points in the source domain associated with the first sensor based on the target domain associated with the second sensor.
5. The autonomous vehicle control system of claim 1 , wherein:
the neural network comprises a plurality of layers; and
the one or more continuous fusion layers are early layers in the plurality of layers.
6. The autonomous vehicle control system of claim 1 , wherein:
the neural network comprises a plurality of layers; and
the one or more continuous fusion layers are late layers in the plurality of layers.
7. The autonomous vehicle control system of claim 1 , wherein the first sensor data comprises camera image data and the second sensor data comprises LIDAR sensor data.
8. The autonomous vehicle control system of claim 7 , the operations further comprising:
generating a mapping between the source domain and the target domain based on a calibration between a camera image sensor configured to generate the camera image data and a LIDAR sensor configured to generate the LIDAR sensor data.
9. The autonomous vehicle control system of claim 1 , wherein processing the first sensor data and the second sensor data with the one or more continuous fusion layers comprises:
receiving a target data point in the target domain associated with the second sensor;
identifying a plurality of source data points in the source domain associated with the first sensor based on the target data point; and
fusing information from the plurality of source data points in the source domain to generate an output feature at the target data point in the target domain.
10. The autonomous vehicle control system of claim 9 , wherein the one or more continuous fusion layers comprise one or more multi-layer perceptrons each having a first portion and a second portion, the first portion of each multi-layer perceptron configured to extract the plurality of source data points from the source domain given the target data point in the target domain, the second portion of each multi-layer perceptron configured to encode an offset between each of the plurality of source data points in the source domain and the target data point in the target domain.
11. An autonomous vehicle, comprising:
one or more sensors that generate sensor data relative to the autonomous vehicle;
one or more processors; and
one or more non-transitory computer-readable media that store instructions for execution by the one or more processors to cause the autonomous vehicle to perform operations, the operations comprising:
providing first sensor data and second sensor data of an environment of the autonomous vehicle to a neural network, wherein the first sensor data is obtained from a first sensor having a first modality, wherein the second sensor data is obtained from a second sensor having a second modality that is different than the first modality, and wherein the neural network comprises one or more continuous fusion layers configured to generate a dense representation of sensor data under a common modality;
processing the first sensor data and the second sensor data with the one or more continuous fusion layers to fuse information from a source domain associated with the first sensor into a target domain associated with the second sensor;
determining a motion plan for controlling a motion of the autonomous vehicle, wherein the motion plan is based at least in part on the fused information; and
controlling the motion of the autonomous vehicle based at least in part on the motion plan.
12. The autonomous vehicle of claim 11 , wherein the dense representation of data points is extracted from both the first modality of the first sensor and the second modality of the second sensor.
13. The autonomous vehicle claim 11 , wherein processing the first sensor data and the second sensor data with the one or more continuous fusion layers is further configured to extract a first portion of data points in the source domain associated with the first sensor based on the target domain associated with the second sensor.
14. The autonomous vehicle of claim 13 , wherein the one or more continuous fusion layers use a nearest neighbor technique to extract the first portion of data points in the source domain associated with the first sensor based on the target domain associated with the second sensor.
15. The autonomous vehicle of claim 11 , wherein the first sensor data comprises camera image data and the second sensor data comprises LIDAR sensor data.
16. The autonomous vehicle of claim 15 , the operations further comprising:
generating a mapping between the source domain and the target domain based on a calibration between a camera image sensor configured to generate the camera image data and a LIDAR sensor configured to generate the LIDAR sensor data.
17. The autonomous vehicle of claim 11 , wherein processing the first sensor data and the second sensor data with the one or more continuous fusion layers comprises:
receiving a target data point in the target domain associated with the second sensor;
identifying a plurality of source data points in the source domain associated with the first sensor based on the target data point; and
fusing information from the plurality of source data points in the source domain to generate an output feature at the target data point in the target domain.
18. The autonomous vehicle of claim 17 , wherein the one or more continuous fusion layers comprise one or more multi-layer perceptrons each having a first portion and a second portion, the first portion of each multi-layer perceptron configured to extract the plurality of source data points from the source domain given the target data point in the target domain, the second portion of each multi-layer perceptron configured to encode an offset between each of the plurality of source data points in the source domain and the target data point in the target domain.
19. A computer-implemented method comprising:
providing first sensor data and second sensor data of an environment of an autonomous vehicle to a neural network, wherein the first sensor data is obtained from a first sensor having a first modality, wherein the second sensor data is obtained from a second sensor having a second modality that is different than the first modality, and wherein the neural network comprises one or more continuous fusion layers configured to generate a dense representation of sensor data under a common modality;
processing the first sensor data and the second sensor data with the one or more continuous fusion layers to fuse information from a source domain associated with the first sensor into a target domain associated with the second sensor;
determining a motion plan for controlling a motion of the autonomous vehicle, wherein the motion plan is based at least in part on the fused information; and
controlling the motion of the autonomous vehicle based at least in part on the motion plan.
20. The computer-implemented method of claim 19 , wherein processing the first sensor data and the second sensor data with the one or more continuous fusion layers comprises:
receiving a target data point in the target domain associated with the second sensor;
identifying a plurality of source data points in the source domain associated with the first sensor based on the target data point; and
fusing information from the plurality of source data points in the source domain to generate an output feature at the target data point in the target domain.