Machine learning based prediction of human interactions with autonomous vehicles
Systems and methods for predicting user interaction with vehicles. A computing device receives an image and a video segment of a road scene, the first at least one of an image and a video segment being taken from a perspective of a participant in the road scene and then generates stimulus data based on the image and the video segment. Stimulus data is transmitted to a user interface and response data is received, which includes at least one of an action and a likelihood of the action corresponding to another participant in the road scene. The computing device aggregates a subset of the plurality of response data to form statistical data and a model is created based on the statistical data. The model is applied to another image or video segment and a prediction of user behavior in the another image or video segment is generated.
1. A computer system comprising:
a computer processor; and
a non-transitory computer readable storage medium storing instructions that when executed by the computer processor cause the computer processor to perform steps of a computer-implemented method for controlling an autonomous vehicle based on a predicted state of mind of road users in a scene captured by a camera of the autonomous vehicle, the steps comprising:
receiving a plurality of images displaying road scenes captured by one or more vehicles;
receiving a plurality of user responses, each user response describing a state of mind of a road user displayed in one or more images;
generating a training dataset comprising summary statistics of uses responses describing the state of minds of road users displayed in the plurality of images;
training, using the training dataset, a supervised learning based model configured to predict summary statistics describing a state of mind of a road user displayed in an input image;
receiving, by an autonomous vehicle, a new image captured by a camera of the autonomous vehicle, the new image of a scene including a road user;
predicting, by the autonomous vehicle, using the supervised learning based model, summary statistics describing a state of mind of the road user in the new image; and
controlling the autonomous vehicle based on the prediction of the supervised learning based model.
2. A computer-implemented method for controlling an autonomous vehicle based on a predicted state of mind of road users in a scene captured by a camera of the autonomous vehicle, the method comprising:
receiving a plurality of images displaying road scenes captured by one or more vehicles;
receiving a plurality of user responses, each user response describing a state of mind of a road user displayed in one or more images;
generating a training dataset comprising summary statistics of user responses describing the state of minds of road users displayed in the plurality of images;
training, using the training dataset, a supervised learning based model configured to predict summary statistics describing a state of mind of a road user displayed in an input image;
receiving, by an autonomous vehicle, a new image captured by a camera of the autonomous vehicle, the new image of a scene including a road user;
predicting, by the autonomous vehicle, using the supervised learning based model, summary statistics describing a state of mind of the road user in the new image; and
controlling the autonomous vehicle based on the prediction of the supervised learning based model.
3. The computer-implemented method of claim 2 , wherein the road user in the input image is a pedestrian and the state of mind predicted by the supervised learning based model indicates whether the pedestrian is likely to perform an action, the action including one of:
staying in place, or
crossing a street.
4. The computer-implemented method of claim 2 , wherein the road user in the input image is a cyclist and the state of mind predicted by the supervised learning based model indicates whether the cyclist is likely to cross a lane.
5. The computer-implemented method of claim 2 , wherein the summary statistics comprises one or more of:
a central tendency,
a variance,
a skew,
a kurtosis,
a scale, or
a histogram.
6. The computer-implemented method of claim 2 , wherein receiving a user response comprises:
sending one or more images for display to a viewer via a user interface, the one or more images including a road user, wherein the user interface is configured to prompt the viewer to describe how the road user will act.
7. The computer-implemented method of claim 2 , wherein receiving a user response comprises:
generating a stimulus comprising a set of images;
sending the generated stimulus for display via a user interface; and
receiving, via the user interface, a user response describing the stimulus.
8. The computer-implemented method of claim 7 , further comprising:
modifying an image to generate a stimulus, the modifying comprising extracting a subset of the image contained in a bounding box around the road user displayed in the image.
9. The computer-implemented method of claim 8 , wherein the supervised learning based model uses one or more features comprising:
dimensions of the bounding box;
location of the bounding box;
shape of the bounding box; or
a change in size or position of the bounding box across two images representing video frames.
10. The computer-implemented method of claim 2 , wherein the user response is determined based on:
an explicit response to a question asked via a user interface; and
an implicit data representing one or more of:
time taken to respond;
whether user deleted keystrokes; or
whether user moved a mouse anywhere other than a location corresponding to a selected response.
11. The computer-implemented method of claim 2 , wherein the supervised learning based model is one of:
a random forest regressor,
a support vector regressor,
a simple neural network,
a deep convolutional neural network,
a recurrent neural network, or
a long short-term memory (LSTM) neural network.
12. A non-transitory computer readable storage medium storing instructions that when executed by a computer processor cause the computer processor to perform steps of a computer-implemented method for controlling an autonomous vehicle based on a predicted state of mind of road users in a scene captured by a camera of the autonomous vehicle, the steps comprising:
receiving a plurality of images displaying road scenes captured by one or more vehicles;
receiving a plurality of user responses, each user response describing a state of mind of a road user displayed in one or more images;
generating a training dataset comprising summary statistics of uses responses describing the state of minds of road users displayed in the plurality of images;
training, using the training dataset, a supervised learning based model configured to predict summary statistics describing a state of mind of a road user displayed in an input image;
receiving, by an autonomous vehicle, a new image captured by a camera of the autonomous vehicle, the new image of a scene including a road user;
predicting, by the autonomous vehicle, using the supervised learning based model, summary statistics describing a state of mind of the road user in the new image; and
controlling the autonomous vehicle based on the prediction of the supervised learning based model.
13. The non-transitory computer readable storage medium of claim 12 , wherein the road user in the input image is a pedestrian and the state of mind predicted by the supervised learning based model indicates whether the pedestrian is likely to perform an action, the action including one of:
staying in place, or
crossing a street.
14. The non-transitory computer readable storage medium of claim 12 , wherein the road user in the input image is a cyclist and the state of mind predicted by the supervised learning based model indicates whether the cyclist is likely to cross a lane.
15. The non-transitory computer readable storage medium of claim 12 , wherein the summary statistics comprises one or more of:
a central tendency,
a variance,
a skew,
a kurtosis,
a scale, or
a histogram.
16. The non-transitory computer readable storage medium of claim 12 , wherein receiving a user response comprises:
sending one or more images for display to a viewer via a user interface, the one or more images including a road user, wherein the user interface is configured to prompt the viewer to describe how the road user will act.
17. The non-transitory computer readable storage medium of claim 12 , wherein receiving a user response comprises:
generating a stimulus comprising a set of images;
sending the generated stimulus for display via a user interface; and
receiving, via the user interface, a user response describing the stimulus.
18. The non-transitory computer readable storage medium of claim 17 , further comprising:
modifying an image to generate a stimulus, the modifying comprising extracting a subset of the image contained in a bounding box around the road user displayed in the image.
19. The non-transitory computer readable storage medium of claim 18 , wherein the supervised learning based model uses one or more features comprising:
dimensions of the bounding box;
location of the bounding box;
shape of the bounding box; or
a change in size or position of the bounding box across two images representing video frames.
20. The non-transitory computer readable storage medium of claim 12 , wherein the user response is determined based on:
an explicit response to a question asked via a user interface; and
an implicit data representing one or more of:
time taken to respond;
whether user deleted keystrokes; or
whether user moved a mouse anywhere other than a location corresponding to a selected response.