Sparse convolutional neural networks
The present disclosure provides systems and methods that apply neural networks such as, for example, convolutional neural networks, to sparse imagery in an improved manner. For example, the systems and methods of the present disclosure can be included in or otherwise leveraged by an autonomous vehicle. In one example, a computing system can extract one or more relevant portions from imagery, where the relevant portions are less than an entirety of the imagery. The computing system can provide the relevant portions of the imagery to a machine-learned convolutional neural network and receive at least one prediction from the machine-learned convolutional neural network based at least in part on the one or more relevant portions of the imagery. Thus, the computing system can skip performing convolutions over regions of the imagery where the imagery is sparse and/or regions of the imagery that are not relevant to the prediction being sought.
1. A computing system for processing imagery captured by one or more sensors of an autonomous vehicle, comprising:
one or more processors;
a machine-learned convolutional neural network; and
one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining the imagery captured by the one or more sensors of the autonomous vehicle;
extracting one or more relevant portions of the imagery, the one or more relevant portions being less than an entirety of the imagery, wherein extracting the one or more relevant portions of the imagery comprises:
generating a binary mask that classifies each of a plurality of sections of the imagery as either sparse or non-sparse; and
determining the one or more relevant portions of the imagery based at least in part on the binary mask;
providing each of the one or more relevant portions of the imagery to the machine-learned convolutional neural network, wherein the machine-learned convolutional neural network performs one or more convolutions respectively on each of the one or more relevant portions; and
receiving a prediction from the machine-learned convolutional neural network based at least in part on the one or more convolutions respectively performed on each of the one or more relevant portions.
2. The computing system of claim 1 , wherein:
the computing system is on-board the autonomous vehicle;
the imagery comprises one or both of:
LIDAR data captured by a LIDAR system mounted on the autonomous vehicle; and
one or more image frames captured by one or more cameras mounted on the autonomous vehicle;
the prediction from the machine-learned convolutional neural network comprises one or both of:
detection of an object in a surrounding environment of the autonomous vehicle; and
a predicted trajectory for the object in the surrounding environment of the autonomous vehicle.
3. The computing system of claim 1 , wherein the operations further comprise controlling motion of the autonomous vehicle based at least in part on the prediction received from the machine-learned convolutional neural network.
4. The computing system of claim 1 , wherein extracting the one or more relevant portions of the imagery comprises:
identifying one or more non-sparse regions of the imagery; and
extracting the one or more relevant portions that respectively correspond to the one or more non-sparse regions.
5. The computing system of claim 1 , wherein generating the binary mask comprises:
dividing the imagery into the plurality of sections;
determining, for each of the plurality of sections, an amount of data included in such section; and
classifying each section as either sparse or non-sparse based at least in part on the amount of data included in such section.
6. The computing system of claim 1 , wherein generating the binary mask comprises:
inputting the imagery into a machine-learned mask generation model; and
receiving the binary mask as an output of the machine-learned mask generation model.
7. The computing system of claim 1 , wherein generating the binary mask comprises:
identifying a region of interest within the imagery;
classifying each section included in the region of interest as non-sparse; and
classifying each section that is not included in the region of interest as sparse.
8. The computing system of claim 7 , wherein identifying the region of interest comprises identifying the region of interest based at least in part on context data associated with an autonomous vehicle.
9. The computing system of claim 8 , wherein the context data associated with the autonomous vehicle comprises a heading of the autonomous vehicle.
10. The computing system of claim 7 , wherein identifying the region of interest comprises identifying the region of interest based at least in part on a confidence metric associated with one or more predictions previously obtained relative to a scene depicted by the imagery.
11. The computing system of claim 1 , wherein determining the one or more relevant portions of the imagery based at least in part on the binary mask comprises:
partitioning the imagery into a plurality of portions, each portion containing two or more of the plurality of sections; and
classifying each portion as either relevant or not relevant based at least in part on the respective classifications of the sections contained in such portion as either sparse or non-sparse.
12. The computing system of claim 1 , wherein:
providing each of the one or more relevant portions of the imagery to the machine-learned convolutional neural network comprises:
stacking the one or more relevant portions in a depth-wise fashion to form a tensor; and
inputting the tensor into the convolutional neural network; and
receiving the prediction from the machine-learned convolutional neural network comprises patching one or more prediction results to the imagery, wherein the one or more prediction results respectively correspond to the one or more relevant portions.
13. The computing system of claim 1 , wherein:
a kernel of the machine-learned convolutional neural network has a kernel size that is larger than at least one of one or more portion sizes respectively associated with at least one relevant portion of the one or more relevant portions of the imagery; and
providing each of the one or more relevant portions of the imagery to the machine-learned convolutional neural network comprises:
identifying at least one kernel portion that respectively corresponds to the at least one relevant portion for which the kernel size is larger than the corresponding portion size; and
computing a layer of the machine-learned convolutional neural network with respect to the at least relevant portion by computing only the identified kernel portion against the at least one relevant portion.
14. The computing system of claim 13 , wherein the machine-learned convolutional neural network performs only valid convolutions in which a kernel size is smaller than a corresponding portion size.
15. The computing system of claim 1 , wherein the machine-learned convolutional neural network includes a residual block and the residual block is configured to provide a sparse update to hidden features.
16. An autonomous vehicle, comprising:
one or more sensors that capture imagery;
one or more processors;
a machine-learned convolutional neural network; and
one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the autonomous vehicle to perform operations, the operations comprising:
obtaining the imagery captured by the one or more sensors of the autonomous vehicle;
extracting one or more relevant portions of the imagery, the one or more relevant portions being less than an entirety of the imagery, wherein extracting the one or more relevant portions of the imagery comprises:
generating a binary mask that classifies each of a plurality of sections of the imagery as either sparse or non-sparse; and
determining the one or more relevant portions of the imagery based at least in part on the binary mask;
providing each of the one or more relevant portions of the imagery to the machine-learned convolutional neural network, wherein the machine-learned convolutional neural network performs one or more convolutions respectively on each of the one or more relevant portions; and
receiving a prediction from the machine-learned convolutional neural network based at least in part on the one or more convolutions respectively performed on each of the one or more relevant portions.