Sparse convolutional neural networks
The present disclosure provides systems and methods that apply neural networks such as, for example, convolutional neural networks, to sparse imagery in an improved manner. For example, the systems and methods of the present disclosure can be included in or otherwise leveraged by an autonomous vehicle. In one example, a computing system can extract one or more relevant portions from imagery, where the relevant portions are less than an entirety of the imagery. The computing system can provide the relevant portions of the imagery to a machine-learned convolutional neural network and receive at least one prediction from the machine-learned convolutional neural network based at least in part on the one or more relevant portions of the imagery. Thus, the computing system can skip performing convolutions over regions of the imagery where the imagery is sparse and/or regions of the imagery that are not relevant to the prediction being sought.
1. A vehicle computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the vehicle computing system to perform operations, the operations comprising:
obtaining first sensor data captured by a first sensor and second sensor data captured by a second sensor;
obtaining first predictions associated with the first sensor data;
generating a region of interest for the second sensor data based on the first predictions;
determining a detection of an object within a surrounding environment of an autonomous vehicle by a machine-learned model and the region of interest for the second sensor data;
determining a predicted trajectory for the object within the surrounding environment based at least in part on the detection of the object; and
controlling motion of the autonomous vehicle based at least in part on the detection of the object or the predicted trajectory for the object within the surrounding environment of the autonomous vehicle.
2. The computing system of claim 1 , wherein the first sensor and the second sensor comprise different modalities.
3. The computing system of claim 1 , wherein determining the detection of the object comprises:
generating a binary mask indicating the region of interest; and
extracting one or more relevant portions of the second sensor data based on the binary mask.
4. The computing system of claim 3 , wherein extracting the one or more relevant portions comprises excluding second sensor data not included in the region of interest by the binary mask.
5. The computing system of claim 3 , wherein the binary mask is used to classify at least one of a plurality of portions of the first sensor data or the second sensor data as either sparse or non-sparse.
6. The computing system of claim 1 , wherein the first predictions comprise at least one detection of an object within the first sensor data.
7. The vehicle computing system of claim 1 , wherein the first sensor comprises one of: (i) a LIDAR system or (ii) one or more cameras; and the second sensor comprises the other of (i) the LIDAR system or (ii) the one or more cameras.
8. The vehicle computing system of claim 7 , wherein the first sensor data comprises at least one of: (i) LIDAR data captured by the LIDAR system or (ii) one or more image frames captured by the one or more cameras; and the second sensor data comprises the other of (i) the LIDAR data captured by the LIDAR system or (ii) the one or more image frames captured by the one or more cameras.
9. The vehicle computing system of claim 1 , wherein the machine-learned model is a convolutional neural network configured to perform one or more convolutions and output at least one of the detection of the object or the predicted trajectory based at least in part on the one or more convolutions.
10. The vehicle computing system of claim 1 , wherein controlling the motion of the autonomous vehicle comprises:
controlling the motion of the autonomous vehicle based at least in part on the detection of the object within the surrounding environment of the autonomous vehicle.
11. The vehicle computing system of claim 1 , wherein controlling the motion of the autonomous vehicle comprises:
controlling the motion of the autonomous vehicle based at least in part on the predicted trajectory for the object.
12. The vehicle computing system of claim 1 , wherein the autonomous vehicle is an autonomous truck.
13. An autonomous vehicle (AV), comprising:
one or more processors; and
one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the vehicle computing system to perform operations, the operations comprising:
obtaining first sensor data captured by a first sensor and second sensor data captured by a second sensor;
obtaining first predictions associated with the first sensor data;
generating a region of interest for the second sensor data based on the first predictions;
determining a detection of an object within a surrounding environment of the autonomous vehicle by a machine-learned model and the region of interest for the second sensor data; and
determining a predicted trajectory for the object within the surrounding environment based at least in part on the detection of the object; and
controlling motion of the autonomous vehicle based at least in part on the detection of the object or the predicted trajectory for the object within the surrounding environment of the autonomous vehicle.
14. The autonomous vehicle of claim 13 , wherein the first sensor and the second sensor comprise different modalities.
15. The autonomous vehicle of claim 13 , wherein determining the detection of the object comprises:
generating a binary mask indicating the region of interest; and
extracting one or more relevant portions of the second sensor data based on the binary mask.
16. The autonomous vehicle of claim 13 , wherein the first sensor comprises one of: (i) a LIDAR system or (ii) one or more cameras; and the second sensor comprises the other of (i) the LIDAR system or (ii) the one or more cameras, and wherein the first sensor data comprises at least one of: (i) LIDAR data captured by the LIDAR system or (ii) one or more image frames captured by the one or more cameras; and the second sensor data comprises the other of (i) the LIDAR data captured by the LIDAR system or (ii) the one or more image frames captured by the one or more cameras.
17. The autonomous vehicle of claim 13 , wherein the machine-learned model is a convolutional neural network configured to perform one or more convolutions and output at least one of the detection of the object or the predicted trajectory based at least in part on the one or more convolutions.
18. The autonomous vehicle of claim 13 , wherein controlling the motion of the autonomous vehicle comprises:
controlling the motion of the autonomous vehicle based at least in part on the detection of the object within the surrounding environment of the autonomous vehicle; and
controlling the motion of the autonomous vehicle based at least in part on the predicted trajectory for the object.
19. The autonomous vehicle of claim 13 , wherein the autonomous vehicle is an autonomous truck.
20. A computer-implemented method for controlling an autonomous vehicle, the computer-implemented method comprising:
obtaining first sensor data captured by a first sensor and second sensor data captured by a second sensor;
obtaining first predictions associated with the first sensor data;
generating a region of interest for the second sensor data based on the first predictions;
determining a detection of an object within a surrounding environment of the autonomous vehicle by a machine-learned model and the region of interest for the second sensor data;
determining a predicted trajectory for the object within the surrounding environment based at least in part on the detection of the object;
controlling motion of the autonomous vehicle based at least in part on the detection of the object within the surrounding environment of the autonomous vehicle; and
controlling motion of the autonomous vehicle based at least in part on the predicted trajectory for the object.