Sparse convolutional neural networks
The present disclosure provides systems and methods that apply neural networks such as, for example, convolutional neural networks, to sparse imagery in an improved manner. For example, the systems and methods of the present disclosure can be included in or otherwise leveraged by an autonomous vehicle. In one example, a computing system can extract one or more relevant portions from imagery, where the relevant portions are less than an entirety of the imagery. The computing system can provide the relevant portions of the imagery to a machine-learned convolutional neural network and receive at least one prediction from the machine-learned convolutional neural network based at least in part on the one or more relevant portions of the imagery. Thus, the computing system can skip performing convolutions over regions of the imagery where the imagery is sparse and/or regions of the imagery that are not relevant to the prediction being sought.
1. A vehicle computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the vehicle computing system to perform operations, the operations comprising:
obtaining sensor data including imagery captured by one or more sensors;
extracting one or more relevant portions of the imagery based at least in part on a determination of one or more portions of the imagery as either sparse or non-sparse, the one or more relevant portions being less than an entirety of the imagery;
determining a detection of an object within a surrounding environment of an autonomous vehicle based at least in part on a machine-learned model and at least one of the one or more relevant portions of the imagery; and
determining a predicted trajectory for the object within the surrounding environment based at least in part on the machine-learned model and the detection of the object.
2. The vehicle computing system of claim 1 , wherein the operations further comprise:
controlling motion of the autonomous vehicle based at least in part on the predicted trajectory for the object.
3. The vehicle computing system of claim 2 , wherein the imagery comprises at least one of: (i) LIDAR data captured by a LIDAR system, or (ii) one or more image frames captured by one or more cameras.
4. The vehicle computing system of claim 1 , wherein extracting the one or more relevant portions of the imagery comprises:
determining the one or more relevant portions of the imagery based at least in part on a binary mask.
5. The vehicle computing system of claim 1 , wherein extracting the one or more relevant portions of the imagery comprises:
identifying one or more non-sparse portions; and
extracting the one or more non-sparse portions.
6. The vehicle computing system of claim 1 , wherein determining the detection of the object within the surrounding environment of the autonomous vehicle based at least in part on the machine-learned model and at least one of the one or more relevant portions of the imagery comprises:
providing the at least one relevant portion of the one or more relevant portions of the imagery to the machine-learned model, wherein the machine-learned model is configured to perform one or more convolutions on the at least one relevant portion.
7. The vehicle computing system of claim 1 , wherein the operations further comprise:
identifying a region of interest within the imagery; and
determining at least one portion of the imagery as either sparse or non-sparse based at least in part on the region of interest within the imagery.
8. The vehicle computing system of claim 7 , wherein identifying the region of interest within the imagery comprises identifying the region of interest based at least in part on context data associated with the autonomous vehicle.
9. The vehicle computing system of claim 8 , wherein the context data associated with the autonomous vehicle comprises at least one of: (i) a heading of the autonomous vehicle, (ii) a trajectory associated with the autonomous vehicle, or (iii) state data associated with the autonomous vehicle, or (iv) state data associated with an object within the surrounding environment.
10. The vehicle computing system of claim 1 , wherein the autonomous vehicle is an autonomous truck.
11. An autonomous vehicle comprising:
one or more sensors;
one or more processors; and
one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the autonomous vehicle to perform operations, the operations comprising:
obtaining sensor data including imagery captured through the one or more sensors;
extracting one or more relevant portions of the imagery based at least in part on a determination of one or more portions of the imagery as either sparse or non-sparse, the one or more relevant portions being less than an entirety of the imagery;
determining a detection of an object within a surrounding environment of the autonomous vehicle based at least in part on a machine-learned model and at least one of the one or more relevant portions of the imagery; and
determining a predicted trajectory for the object within the surrounding environment based at least in part on the detection of the object.
12. The autonomous vehicle of claim 11 , wherein the one or more sensors comprise at least one of: (i) a LIDAR system or (ii) one or more cameras.
13. The autonomous vehicle of claim 11 , wherein the operations further comprises:
determining motion for the autonomous vehicle based at least in part on the object within the surrounding environment.
14. The autonomous vehicle of claim 11 , wherein the operations further comprises:
determining motion for the autonomous vehicle based at least in part on the predicted trajectory of the object within the surrounding environment.
15. The autonomous vehicle of claim 11 , wherein extracting the one or more relevant portions of the imagery comprises:
determining, based at least in part on a binary mask, the one or more relevant portions of the imagery.
16. The autonomous vehicle of claim 15 , wherein the binary mask is used to classify at least one of a plurality of portions as either sparse or non-sparse.
17. The autonomous vehicle of claim 11 , wherein extracting the one or more relevant portions of the imagery comprises:
extracting the one or more relevant portions that respectively correspond to one or more non-sparse regions of the imagery.
18. The autonomous vehicle of claim 11 , wherein the machine-learned model is a convolutional neural network configured to perform one or more convolutions on the at least one relevant portion and output at least one of the detection of the object or the predicted trajectory based at least in part on the one or more convolutions performed on the at least one relevant portion.
19. A computer-implemented method comprising:
obtaining imagery captured by one or more sensors of an autonomous vehicle;
extracting one or more relevant portions of the imagery based at least in part on a determination of one or more portions of the imagery as either sparse or non-sparse, the one or more relevant portions being less than an entirety of the imagery;
determining a detection of an object within a surrounding environment of the autonomous vehicle based at least in part on a machine-learned model and at least one of the one or more relevant portions of the imagery; and
determining a predicted trajectory for the object within the surrounding environment based at least in part on the machine-learned model and the detection of the object.
20. The computer-implemented method of claim 19 , wherein the determination of the one or more portions of the imagery as either sparse or non-sparse is based at least in part on a region of interest within the imagery.