Multi-resolution top-down prediction
Techniques for determining a classification probability of an object in an environment are discussed herein. Techniques may include analyzing sensor data associated with an environment from a perspective, such as a top-down perspective, using multi-channel data. From this perspective, techniques may determine channels of multi-channel input data and additional feature data. Channels corresponding to spatial features may be included in the multi-channel input data and data corresponding to non-spatial features may be included in the additional feature data. The multi-channel input data may be input to a first portion of a machine-learned (ML) model, and the additional feature data may be concatenated with intermediate output data from the first portion of the ML model, and input into a second portion of the ML model for subsequent processing and to determine the classification probabilities. Additionally, techniques may be performed on a multi-resolution voxel space representing the environment.
1. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
receiving sensor data from a sensor associated with an environment including an autonomous vehicle;
determining, based at least in part on the sensor data, first multi-channel input data representing a first top-down view of a first area of the environment;
determining, based at least in part on the sensor data, second multi-channel input data representing a second top-down view of a second area of the environment, the second area of the environment being greater than the first area and the second multi-channel input data having a lower resolution than the first multi-channel input data;
inputting the first multi-channel input data into a first machine-learned (ML) model;
inputting the second multi-channel input data into a second ML model;
receiving first output data from the first ML model, the first output data comprising a first classification associated with an object in the environment;
receiving second output data from the second ML model, the second output data comprising a second classification associated with the object; and
controlling the autonomous vehicle based at least in part on the first classification or the second classification.
2. The system of claim 1 , wherein the first area of the environment at least partially overlaps with the second area of the environment.
3. The system of claim 1 , the operations further comprising:
determining that the object is associated with the first area of the environment; and
discarding the second classification based at least on determining that the object is within the first area;
wherein controlling the autonomous vehicle is based at least in part on the first classification.
4. The system of claim 1 , wherein a first channel of the first multi-channel input data comprises first feature data representing a feature associated with the object, and a second channel of the second multi-channel input data comprises second feature data representing the feature associated with the object.
5. The system of claim 1 , wherein the first multi-channel input data represents a spatial feature and the object in the environment.
6. A method comprising:
receiving sensor data from a sensor associated with an environment;
determining, based at least in part on the sensor data, first input data representing a first area of the environment;
determining, based at least in part on the sensor data, second input data representing a second area of the environment, the second area of the environment being greater than the first area and the second input data representing the environment at a lower resolution than the first input data;
inputting the first input data into a first machine-learned (ML) model;
inputting the second input data into a second ML model; and
controlling an autonomous vehicle based at least in part on a first classification associated with an object in the environment received from the first ML model, or a second classification associated with the object in the environment received from the second ML model.
7. The method of claim 6 , wherein the first area of the environment at least partially overlaps with the second area of the environment.
8. The method of claim 6 , further comprising:
determining that the object is associated with the first area of the environment; and
discarding the second classification based at least on determining that the object is within the first area;
wherein controlling the autonomous vehicle is based at least in part on the first classification.
9. The method of claim 6 , wherein:
the first input data comprises first multi-channel input data;
a first channel of the first multi-channel input data comprises first feature data representing a feature associated with the object; and
a second channel of second multi-channel input data comprises second feature data representing the feature associated with the object.
10. The method of claim 6 , wherein the first input data represents a spatial feature and the object in the environment.
11. The method of claim 6 , further comprising determining feature data associated with the sensor data, the feature data representing features associated with the object, and wherein the first input data comprises a number of channels based at least in part on a first subset of the features.
12. The method of claim 11 , further comprising:
inputting the first input data into a first portion of the first ML model;
receiving intermediate output data from the first portion of the first ML model;
determining, based at least in part on a second subset of the features associated with the object and the intermediate output data, intermediate input data;
inputting the intermediate input data into a second portion of the first ML model; and
receiving, from the first portion of the first ML model, first output data comprising the first classification associated with the object.
13. The method of claim 11 , wherein:
the first subset of the features comprises at least one of:
map information associated with the environment;
a bounding box associated with the object; or
a size associated with the object; and
a second subset of the features comprises at least one of:
a velocity associated with the object;
an acceleration associated with the object; or
a lighting state associated with the object.
14. The method of claim 6 , further comprising associating the sensor data with a three-dimensional voxel space representing the environment, the three-dimensional voxel space including the first area and the second area; and
wherein the first input data includes voxel data corresponding to the first area within the three-dimensional voxel space.
15. One or more non-transitory computer-readable media storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising:
receiving sensor data from a sensor associated with an environment;
determining, based at least in part on the sensor data, first input data representing a first area of the environment;
determining, based at least in part on the sensor data, second input data representing a second area of the environment, the second area of the environment being greater than the first area and the second input data representing the environment at a lower resolution than the first input data;
inputting the first input data into a first machine-learned (ML) model;
inputting the second input data into a second ML model; and
controlling an autonomous vehicle based at least in part on a first classification associated with an object in the environment received from the first ML model, or a second classification associated with the object in the environment received from the second ML model.
16. The one or more non-transitory computer-readable media of claim 15 , wherein the first area of the environment at least partially overlaps with the second area of the environment.
17. The one or more non-transitory computer-readable media of claim 16 , the operations further comprising:
determining that the object is associated with the first area of the environment; and
discarding the second classification based at least on determining that the object is within the first area;
wherein controlling the autonomous vehicle is based at least in part on the first classification.
18. The one or more non-transitory computer-readable media of claim 15 , wherein:
the first input data comprises first multi-channel input data;
a first channel of the first multi-channel input data comprises first feature data representing a feature associated with the object; and
a second channel of second multi-channel input data comprises second feature data representing the feature associated with the object.
19. The one or more non-transitory computer-readable media of claim 15 , wherein the first input data represents a spatial feature and the object in the environment.
20. The one or more non-transitory computer-readable media of claim 15 , the operations further comprising determining feature data associated with the sensor data, the feature data representing features associated with the object, and wherein the first input data comprises a number of channels based at least in part on a first subset of the features.