Intermediate input for machine learned model
Techniques for determining a classification probability of an object in an environment are discussed herein. Techniques may include analyzing sensor data associated with an environment from a perspective, such as a top-down perspective, using multi-channel data. From this perspective, techniques may determine channels of multi-channel input data and additional feature data. Channels corresponding to spatial features may be included in the multi-channel input data and data corresponding to non-spatial features may be included in the additional feature data. The multi-channel input data may be input to a first portion of a machine-learned (ML) model, and the additional feature data may be concatenated with intermediate output data from the first portion of the ML model, and input into a second portion of the ML model for subsequent processing and to determine the classification probabilities. Additionally, techniques may be performed on a multi-resolution voxel space representing the environment.
1. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
receiving sensor data from a sensor associated with an environment including an object;
determining, based at least in part on the sensor data, multi-channel image data representing a top-down view of the environment, the multi-channel image data including image data associated with a spatial feature, wherein the spatial feature comprises map information associated with the environment and at least one of a location of the object in the environment, a bounding box associated with the object, or a size associated with the object;
inputting the multi-channel image data into a first portion of a machine learned (ML) model;
receiving, as an intermediate output, intermediate output data from the first portion of the ML model wherein the intermediate output data includes the spatial feature encoded as being associated with the object;
determining, based at least in part on the sensor data, non-spatial feature data representing a non-spatial feature associated with the object;
concatenating, as concatenated data, the intermediate output data with the non-spatial feature data on a per object basis;
inputting the concatenated data into a second portion of the ML model;
receiving output data from the second portion of the ML model; and
determining a classification probability for the object based at least in part on the output data.
2. The system of claim 1 , wherein:
the non-spatial feature comprises at least one of:
a velocity associated with the object;
an acceleration associated with the object; or
a lighting state associated with the object.
3. The system of claim 1 , wherein the classification probability comprises at least one of:
an object type classification;
an object behavior classification;
an object gaze classification;
an object trajectory classification;
a lane change classification; or
an emergency vehicle classification.
4. The system of claim 1 , further comprising:
determining a graphical reference corresponding to a corresponding location of the object within the multi-channel image data, wherein the spatial feature is encoded as being associated with the object based at least in part on the graphical reference.
5. The system of claim 1 , wherein the first portion of the ML model comprises a Convolutional Neural Network (CNN) and the second portion of the ML model comprises a Deep Neural Network (DNN).
6. A method comprising:
receiving sensor data from a sensor associated with an environment including an object;
determining, based at least in part on the sensor data, spatial data representing the environment;
inputting the spatial data into a first portion of a machine learned (ML) model, wherein the spatial data comprises map information associated with the environment and at least one of a location of the object in the environment, a bounding box associated with the object, or a size associated with the object;
receiving intermediate data from the first portion of the ML model, wherein the intermediate data includes spatial feature data corresponding to a spatial feature encoded as being associated with the object;
determining, based at least in part on the sensor data, secondary feature data corresponding to a feature associated with the object;
concatenating, as concatenated data, the intermediate data with the secondary feature data on a per object basis;
inputting the concatenated data into a second portion of the ML model; and
determining a classification probability of the object based at least in part on data received from the second portion of the ML model.
7. The method of claim 6 , wherein:
the feature comprises at least one of:
a velocity associated with the object;
an acceleration associated with the object; or
a lighting state associated with the object.
8. The method of claim 7 , wherein the spatial data represents a top-down view of the environment.
9. The method of claim 8 , further comprising:
determining a mask identifying a corresponding location of the object in the spatial data; and
determining, based on the mask and the spatial data, a feature vector associated with the object, wherein the intermediate data includes the feature vector.
10. The method of claim 6 , wherein the first portion of the ML model comprises a Convolutional Neural Network (CNN) and the second portion of the ML model comprises a Deep Neural Network (DNN).
11. The method of claim 6 , wherein the spatial data is first spatial data corresponding to a first range to an origin in the environment, wherein the ML model comprises a first ML model, and the method further comprising:
determining, based at least in part on the sensor data, second spatial data, wherein the spatial data corresponds to a second range to the origin, the first range being less than the second range;
inputting the second spatial data into a second ML model configured to determine classification probabilities associated with the second range; and
wherein determining the classification probability is further based at least in part on an output of the second ML model.
12. The method of claim 6 , further comprising:
associating the sensor data with a three-dimensional voxel space representing the environment; and
wherein the spatial data represents a reduced-dimensionality representation of the three-dimensional voxel space.
13. The method of claim 6 , further comprising receiving the sensor data from a sensor associated with an autonomous vehicle in the environment.
14. The method of claim 6 , further comprising controlling an autonomous vehicle based at least in part on the classification probability.
15. One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
receiving sensor data from a sensor associated with an environment including an object;
determining, based at least in part on the sensor data, spatial data representing the environment;
inputting the spatial data into a first portion of a machine learned (ML) model, wherein the spatial data comprises map information associated with the environment and at least one of a location of the object in the environment, a bounding box associated with the object, or a size associated with the object;
receiving intermediate data from the first portion of the ML model, wherein the intermediate data includes spatial feature data corresponding to a spatial feature encoded as being associated with the object;
determining, based at least in part on the sensor data, secondary feature data corresponding to a feature associated with the object;
concatenating, as concatenated data, the intermediate data with the secondary data on a per object basis;
inputting the concatenated data into a second portion of the ML model; and
determining a classification probability of the object based at least in part on data received from the second portion of the ML model.
16. The one or more non-transitory computer-readable media of claim 15 , wherein:
the feature comprises at least one of:
a velocity associated with the object;
an acceleration associated with the object; or
a lighting state associated with the object.
17. The one or more non-transitory computer-readable media of claim 16 , wherein the spatial data represents a top-down view of the environment.
18. The one or more non-transitory computer-readable media of claim 17 , wherein the spatial data includes:
a first channel comprising a first spatial feature; and
a second channel comprising a second spatial feature.
19. The one or more non-transitory computer-readable media of claim 15 , the operations further comprising:
determining a mask identifying a corresponding location of the object in the spatial data; and
determining based on the mask and the spatial data, a feature vector associated with the object, wherein the intermediate data includes the feature vector.
20. The one or more non-transitory computer-readable media of claim 15 , wherein the first portion of the ML model comprises a Convolutional Neural Network (CNN) and the second portion of the ML model comprises a Deep Neural Network (DNN).