System and method for determining object intention through visual attributes
Systems and methods for determining object intentions through visual attributes are provided. A method can include determining, by a computing system, one or more regions of interest. The regions of interest can be associated with surrounding environment of a first vehicle. The method can include determining, by a computing system, spatial features and temporal features associated with the regions of interest. The spatial features can be indicative of a vehicle orientation associated with a vehicle of interest. The temporal features can be indicative of a semantic state associated with signal lights of the vehicle of interest. The method can include determining, by the computing system, a vehicle intention. The vehicle intention can be based on the spatial and temporal features. The method can include initiating, by the computing system, an action. The action can be based on the vehicle intention.
1. A computer-implemented method of determining semantic vehicle intentions, comprising:
obtaining sensor data associated with a surrounding environment of a first vehicle, wherein the sensor data comprises a sequence of image frames respectively corresponding to one of a plurality of time steps;
determining one or more regions of interest associated with the sensor data;
determining one or more spatial features associated with at least one of the one or more regions of interest, wherein at least one of the one or more spatial features are indicative of a vehicle orientation associated with a second vehicle;
determining one or more temporal features associated with at least one of the one or more regions of interest, wherein the one or more temporal features are indicative of one or more semantic states associated with at least one signal light of the second vehicle;
determining, using one or more machine learned models, an intention indicative of a predicted future action of the second vehicle based on the one or more spatial features and the one or more temporal features, wherein the one or more spatial features and the one or more temporal features are input into at least one of the one or more machine learned models; and
initiating one or more actions based on the intention.
2. The computer-implemented method of claim 1 , wherein the one or more regions of interest comprise one or more cropped image frames associated with the second vehicle.
3. The computer-implemented method of claim 2 , wherein the one or more cropped image frames comprise data indicative of the at least one signal light of the second vehicle.
4. The computer-implemented method of claim 1 , wherein at least one of the one or more spatial features are indicative of a model representation of the second vehicle.
5. The computer-implemented method of claim 4 , wherein the model representation of the second vehicle is indicative of a vehicle orientation associated with the second vehicle.
6. The computer-implemented method of claim 5 , wherein the vehicle orientation associated with the second vehicle is relative to the first vehicle.
7. The computer-implemented method of claim 1 , wherein at least one of the one or more spatial features are indicative of one or more occluding objects.
8. The computer-implemented method of claim 7 , wherein the one or more occluding objects comprise one or more objects disrupting a view of the second vehicle from the first vehicle.
9. The computer-implemented method of claim 1 , wherein the one or more actions comprise providing one or more informational prompts to an operator of the first vehicle.
10. A computing system comprising:
one or more processors; and
one or more tangible, non-transitory, computer readable media that store instructions that when executed by the one or more processors cause the computing system to perform operations comprising:
obtaining sensor data associated with a surrounding environment of a first vehicle;
determining, via one or more machine learning models, one or more regions of interest associated with the sensor data;
determining, via the one or more machine learning models, one or more spatial features associated with at least one of the one or more regions of interest, wherein at least one of the one or more spatial features are indicative of an object orientation associated with an object of interest;
determining, via the one or more machine learning models, one or more temporal features associated with at least one of the one or more regions of interest, wherein the one or more temporal features are indicative of one or more semantic states associated with at least one signal of the object of interest;
determining, via the one or more machine learning models, an intention indicative of a predicted future action of the object of interest based on the one or more spatial features and the one or more temporal features; and
initiating one or more actions based on the intention.
11. The computing system of claim 10 , wherein the sensor data comprises a sequence of image frames at a plurality of time steps.
12. The computing system of claim 11 , wherein determining the one or more regions of interest comprises inputting the sequence of image frames into at least one of the one or more machine learning models.
13. The computing system of claim 10 , wherein determining the one or more spatial features comprises inputting the one or more regions of interest into at least one of the one or more machine learning models.
14. The computing system of claim 10 , wherein determining the one or more temporal features comprises inputting a series of the one or more regions of interest into at least one of the one or more machine learning models.
15. The computing system of claim 10 , wherein determining the intention associated with the object of interest comprises inputting the one or more spatial features and the one or more temporal features into at least one of the one or more machine learning models.
16. The computing system of claim 10 , wherein the regions of interest, spatial features, temporal features, and object intention are respectively determined separately by a different model of the one or more machine learning models.
17. An autonomous vehicle comprising:
one or more vehicle sensors;
one or more processors; and
one or more tangible, non-transitory, computer readable media that store instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising:
obtaining, via the one or more vehicle sensors, sensor data associated with a surrounding environment of the autonomous vehicle, wherein the sensor data comprises a sequence of image frames at a plurality of time steps;
determining a region of interest associated with the sensor data;
determining one or more spatial features associated with the one or more region of interest, wherein at least one of the one or more spatial features are indicative of a second vehicle orientation;
determining one or more temporal features associated with the region of interest;
determining, using one or more machine learned models, an intention indicative of a predicted future action of the second vehicle based on the one or more spatial features and the one or more temporal features, wherein the one or more spatial features and the one or more temporal features are input into at least one of the one or more machine learned models; and
initiating one or more actions based on the intention.
18. The autonomous vehicle of claim 17 , wherein the autonomous vehicle further comprises one or more output devices, and wherein the operations further comprise:
providing, via the one or more output devices, data indicative of the intention to one or more operators of the autonomous vehicle.
19. The autonomous vehicle of claim 17 , wherein the one or more actions comprise generating a motion plan for the autonomous vehicle based on the intention.
20. The autonomous vehicle of claim 19 , wherein the one or more actions further comprise initiating one or more actions based on the motion plan.