Hyper planning based on object and/or region
A vehicle computing system may implement techniques to predict behavior of objects detected by a vehicle operating in the environment. The techniques may include determining a feature with respect to a detected objects (e.g., likelihood that the detected object will impact operation of the vehicle) and/or a location of the vehicle and determining based on the feature a model to use to predict behavior (e.g., estimated states) of proximate objects (e.g., the detected object). The model may be configured to use one or more algorithms, classifiers, and/or computational resources to predict the behavior. Different models may be used to predict behavior of different objects and/or regions in the environment. Each model may receive sensor data as an input, and output predicted behavior for the detected object. Based on the predicted behavior of the object, a vehicle computing system may control operation of the vehicle.
1. A non-transitory computer-readable storage media storing instructions that, when executed, cause one or more processors to perform operations comprising:
detecting an object in an environment around a vehicle based at least in part on sensor data;
determining a type of feature associated with the object or the vehicle based at least in part on the sensor data or map data;
determining a first model for the object based at least in part on the type of feature;
predicting, based at least in part on the first model, a first behavior of the object;
determining a second model from among multiple available models based at least in part on the type of feature associated with the object;
predicting, by the second model, a second behavior of the object; and
controlling operation of the vehicle based at in part on the second behavior from the second model.
2. The non-transitory computer-readable storage media of claim 1 , wherein the type of feature indicates a relevance of the object to operation of the vehicle.
3. The non-transitory computer-readable storage media of claim 1 , wherein the type of feature comprises at least one of:
a semantic classification of the object;
an attribute of the object;
an attribute of the environment;
a proximity of the object relative to the vehicle;
an attribute of the vehicle relative to another object in the environment;
a location of the object;
a velocity of the object relative to a velocity of the vehicle;
a route of the vehicle to a destination in the environment;
a context of the object;
a region adjacent to the vehicle;
a likelihood that the object impacts operation of the vehicle;
an interaction between the object and the vehicle; or
an interaction between objects in the environment.
4. The non-transitory computer-readable storage media of claim 1 , wherein the second model is more computationally intensive than the first model.
5. The non-transitory computer-readable storage media of claim 1 , wherein the first model comprises a first machine learned model trained to output a classification of the object and the second model comprises a second machine learned model trained to recognize a gesture associated with the object.
6. The non-transitory computer-readable storage media of claim 1 , the operations further comprising:
inputting at least a portion of the sensor data into the second model; and
receiving, from the second model, second output indicative of the second behavior of the object;
wherein controlling operation of the vehicle is further based at least in part on the second behavior being associated with a finer level of movement detection with respect to the first behavior of the object.
7. The non-transitory computer-readable storage media of claim 1 , the operations further comprising sending an instruction to at least one of:
a sensor of the vehicle to change at least one of a resolution, a bit rate, a rate of capture, or a compression, at which sensor data is captured or stored; or
a perception component to change an input of the perception component.
8. A method comprising:
receiving sensor data from a sensor associated with a vehicle traversing an environment;
determining, based at least in part on one or more of the sensor data or map data, a type of feature associated with the environment;
determining, based at least in part on the type of feature, a first model;
inputting at least a portion of the sensor data into the first model;
receiving, from the first model, a first output indicative of a behavior of an object in the environment;
determining a second model from among multiple available models based at least in part on the feature;
inputting at least a portion of the sensor data into the second model;
receiving, from the second model, a second output indicative of another behavior of the object; and
controlling operation of the vehicle based at in part on the second output of the second model.
9. The method of claim 8 , wherein the type of feature associated with the environment comprises at least one of: an attribute of the environment, an attribute of the object in the environment, map data, an occluded region, or a region of interest.
10. The method of claim 8 , wherein the type of feature comprises a relevance of the object to operation of the vehicle.
11. The method of claim 8 , wherein predicting the behavior of the object comprises determining an attribute of the object or a location of the object in the environment at a future time.
12. The method of claim 8 , further comprising sending an instruction to at least one of:
a sensor of the vehicle to change at least one of a resolution, a bit rate, a rate of capture, a compression, at which sensor data is captured or stored; or
a perception component to change an input of the perception component.
13. The method of claim 8 , wherein:
the first model is trained to output at least one of a detection of the object or a classification of the object; and
the second model is trained to output at least one of a velocity of the object, a predicted trajectory of the object, a pose of the object, a subclassification of the object, a detection of a portion of the object, a detection of a face, or a detection of a gesture.
14. The method of claim 8 , wherein the first model comprises a machine learned model trained to output a classification of the object and the second model is not a machine learned model.
15. A system comprising:
a sensor associated with a vehicle traversing an environment;
one or more processors; and
non-transitory computer-readable storage media storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
detecting an object in an environment around a vehicle based at least in part on sensor data;
determining a type of feature associated with the object or the vehicle based at least in part on the sensor data or map data;
determining a first model for the object based at least in part on the type of feature;
predicting, based at least in part on the first model, a first behavior of the object;
determining a second model from among multiple available models based at least in part on the type of feature associated with the object;
predicting, by the second model, a second behavior of the object; and
controlling operation of the vehicle based at in part on the second behavior from the second model.
16. The system of claim 15 , wherein the type of feature indicates a relevance of the object to operation of the vehicle.
17. The system of claim 15 , wherein the type of feature comprises at least one of:
a semantic classification of the object;
an attribute of the object;
an attribute of the environment;
a proximity of the object relative to the vehicle;
an attribute of the vehicle relative to another object in the environment;
a location of the object;
a velocity of the object relative to a velocity of the vehicle;
a route of the vehicle to a destination in the environment;
a context of the object;
a region adjacent to the vehicle;
a likelihood that the object impacts operation of the vehicle;
an interaction between the object and the vehicle; or
an interaction between objects in the environment.
18. The system of claim 15 , wherein the second model is more computationally intensive than the first model.
19. The system of claim 15 , wherein the first model comprises a first machine learned model trained to output a classification of the object and the second model comprises a second machine learned model trained to recognize a gesture associated with the object.
20. The system of claim 15 , the operations further comprising:
inputting at least a portion of the sensor data into the second model; and
receiving, from the second model, second output indicative of behavior of the object;
wherein controlling operation of the vehicle is further based at least in part on the second behavior being associated with a finer level of movement detection with respect to the first behavior of the object.