Inferring intent using computer vision
A system trains a model to infer an intent of an entity. The model includes one or more sensors to obtain frames of data, one or more processors, and a memory storing instructions that, when executed by the one or more processors, cause the system to perform steps. A first step includes determining, in each frame of the frames, one or more bounding regions, each of the bounding regions enclosing an entity. A second step includes identifying a common entity, the common entity being present in bounding regions corresponding to a plurality of the frames. A third step includes associating the common entity across the frames. A fourth step includes training a model to infer an intent of the common entity based on data outside of the bounding regions.
1 . A system configured to train a model to infer an intent of an entity, comprising:
one or more sensors configured to obtain frames of data;
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the system to perform:
determining, in each frame of the frames, one or more bounding regions, each of the bounding regions enclosing an entity; and
inferring, based on a trained model, an intent associated with the entity based on data outside of the bounding regions, wherein inferring an intent is based on semantic segmentation to predict a category or a classification associated with one or more pixels of the frames and based on instance segmentation to predict whether two pixels associated with a common category or a common classification belong to same or different instances.
2 . The system of claim 1 , wherein the trained model is trained based on Lidar data.
3 . The system of claim 1 , wherein the instructions further cause the system to perform:
rescaling a segmentation output to fit dimensions of the bounding regions.
4 . The system of claim 1 , wherein:
the one or more sensors comprise a camera;
the entity comprises a vehicle; and
the intent is associated with a turning or braking maneuver of the vehicle.
5 . The system of claim 4 , wherein the intent is associated with a left or right turn signal.
6 . The system of claim 1 , wherein the trained model infers an intent based on a probability of a left turn signal of a vehicle being on, a probability of a right turn signal of the vehicle being on, and a probability of a brake light of the vehicle being on.
7 . The system of claim 4 , wherein the trained model is trained based on cross entropy losses over the inferred intent, over left or right turn signals of the vehicle, and over the vehicle.
8 . The system of claim 1 , wherein the inferring of the intent comprises inferring the intent under different weather and lighting conditions.
9 . The system of claim 1 , wherein the trained model is trained based on a classification loss, a bounding box loss, and a mask prediction loss.
10 . The system of claim 1 , wherein the model is associated with a softmax layer that determines probabilities that each pixel of the frames belongs to a particular classification or category.
11 . A method comprising:
obtaining, using one or more sensors, frames of data;
determining, in each frame of the frames, one or more bounding regions, each of the bounding regions enclosing an entity; and
inferring, based on a trained model, an intent associated with the entity based on data outside of the bounding regions, wherein inferring an intent is based on semantic segmentation to predict a category or a classification associated with one or more pixels of the frames and based on instance segmentation to predict whether two pixels associated with a common category or a common classification belong to same or different instances.
12 . A system configured to train a model to infer an intent of an entity, comprising:
one or more sensors configured to obtain frames of data;
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the system to perform:
determining, in each frame of the frames, one or more bounding regions, each of the bounding regions enclosing an entity; and
inferring, based on a trained model, an intent associated with the entity based on data outside of the bounding regions, wherein inferring an intent is based on a probability of a left turn signal of a vehicle being on, a probability of a right turn signal of the vehicle being on, and a probability of a brake light of the vehicle being on, wherein the trained model is trained based on cross entropy losses over the inferred intent, over left or right turn signals of the vehicle, and over the vehicle.