System and method for determining object intention through visual attributes
Systems and methods for determining object intentions through visual attributes are provided. A method can include determining, by a computing system, one or more regions of interest. The regions of interest can be associated with surrounding environment of a first vehicle. The method can include determining, by a computing system, spatial features and temporal features associated with the regions of interest. The spatial features can be indicative of a vehicle orientation associated with a vehicle of interest. The temporal features can be indicative of a semantic state associated with signal lights of the vehicle of interest. The method can include determining, by the computing system, a vehicle intention. The vehicle intention can be based on the spatial and temporal features. The method can include initiating, by the computing system, an action. The action can be based on the vehicle intention.
1 . A computer-implemented method, comprising:
obtaining sensor data associated with a surrounding environment of a first vehicle, wherein the sensor data comprises a sequence of image frames respectively corresponding to a plurality of time steps,
determining one or more regions of interest associated with the sensor data;
determining a model representation of an object associated with at least one of the one or more regions of interest, wherein the model representation indicates physical characteristics associated with the object, the physical characteristics comprising an orientation of the object within the surrounding environment of the first vehicle;
determining one or more features indicative of one or more states associated with at least one signal of the object;
determining an intention indicative of a predicted future action of the object based on the model representation, the physical characteristics, and the one or more features,
generating a trajectory of the first vehicle based on the intention, wherein the trajectory is responsive to the predicted future action of the object; and
initiating one or more actions to control the first vehicle according to the trajectory.
2 . The computer-implemented method of claim 1 , wherein the object is a second vehicle.
3 . The computer-implemented method of claim 2 , wherein the signal of the object is at least one signal light of the second vehicle, and wherein at least one of the one or more features is a temporal feature indicative of one or more states associated with the at least one signal light of the second vehicle.
4 . The computer-implemented method of claim 1 , wherein the object is a bicycle.
5 . The computer-implemented method of claim 4 , wherein the signal comprises a hand wave by an operator of the bicycle.
6 . The computer-implemented method of claim 1 , wherein determining the intention indicative of the predicted future action of the object comprises determining the intention using a machine learned model, wherein the one or more features are input into the machine learned model.
7 . The computer-implemented method of claim 1 , wherein the one or more regions of interest comprise one or more cropped image frames associated with the object.
8 . The computer-implemented method of claim 7 , wherein the one or more cropped image frames comprise data indicative of the at least one signal of the object.
9 . A computing system comprising:
one or more processors;
one or more tangible, non-transitory, computer readable media that store instructions that are executable by the one or more processors to perform operations comprising:
obtaining sensor data associated with a surrounding environment of a first vehicle, wherein the sensor data comprises a sequence of image frames respectively corresponding to a plurality of time steps;
determining one or more regions of interest associated with the sensor data;
determining a model representation of an object associated with at least one of the one or more regions of interest, wherein the model representation indicates physical characteristics associated with the object, the physical characteristics comprising an orientation of the object within the surrounding environment of the first vehicle;
determining one or more features indicative of one or more states associated with at least one signal of the object;
determining an intention indicative of a predicted future action of the object based on the model representation, the physical characteristics, and the one or more features;
generating a trajectory of the first vehicle based on the intention, wherein the trajectory is responsive to the predicted future action of the object; and
initiating one or more actions to control the first vehicle according to the trajectory.
10 . The computing system of claim 9 , wherein determining the one or more features comprise inputting a series of images of the one or more regions of interest into a machine learning model.
11 . The computing system of claim 9 , wherein determining the intention associated with the object is inputting the one or more features into a machine learning model.
12 . The computing system of claim 9 , wherein the model representation indicates an orientation of the object and wherein the orientation of the object is relative to an additional object within a surrounding environment of the first vehicle.
13 . The computing system of claim 9 , wherein determining the one or more regions of interest comprise inputting the sequence of image frames into one or more machine learning models.
14 . The computing system of claim 9 , wherein the features and the intention are respectively determined by different machine learned models of one or more machine learned models.
15 . A vehicle comprising, comprising:
one or more processors; and
one or more tangible, non-transitory, computer readable media that store instructions that are executable by the one or more processors to perform operations comprising:
obtaining sensor data associated with a surrounding environment of the vehicle, wherein the sensor data comprises a sequence of image frames respectively corresponding to one of a plurality of time steps;
determining one or more regions of interest associated with the sensor data;
determining a model representation of an object associated with at least one of the one or more regions of interest, wherein the model representation indicates physical characteristics associated with the object, the physical characteristics comprising an orientation of the object within the surrounding environment of the vehicle;
determining one or more features associated with at least one of the one or more regions of interest, wherein the one or more features are indicative of one or more states associated with at least one signal of the object;
determining an intention indicative of a predicted future action of the object based on the model representation, the physical characteristics, and the one or more features;
generating a trajectory of the vehicle based on the intention, wherein the trajectory is responsive to the predicted future action of the object; and
initiating one or more actions to control the vehicle according to the trajectory.
16 . The vehicle of claim 15 , wherein the vehicle is an autonomous vehicle.
17 . The vehicle of claim 15 , wherein the operations further comprise:
generating one or more motion plans for the vehicle based on the intention.
18 . The vehicle of claim 17 , wherein initiating the one or more actions is based on at least one motion plan of the one or more motion plans.
19 . The vehicle of claim 15 , wherein the vehicle further comprises one or more output devices, and wherein the operations further comprise:
providing, via the one or more output devices, data indicative of the intention to one or more operators of the vehicle.
20 . The vehicle of claim 15 , wherein determining the intention associated with the object is inputting the one or more features into a machine learning model.