System and method of predicting human interaction with vehicles
Systems and methods for predicting user interaction with vehicles. A computing device receives an image and a video segment of a road scene, the first at least one of an image and a video segment being taken from a perspective of a participant in the road scene and then generates stimulus data based on the image and the video segment. Stimulus data is transmitted to a user interface and response data is received, which includes at least one of an action and a likelihood of the action corresponding to another participant in the road scene. The computing device aggregates a subset of the plurality of response data to form statistical data and a model is created based on the statistical data. The model is applied to another image or video segment and a prediction of user behavior in the another image or video segment is generated.
1. A computer-implemented method for controlling a moveable device, the method comprising:
receiving a plurality of training images, wherein each image from the plurality of training images displays one or more persons;
for each of the plurality of training images:
sending the training image to one or more annotators via a user interface requesting user responses describing a state of mind of a person displayed in the training image; and
receiving user responses from the annotators, each user response describing the state of mind of the one or more persons displayed in the image;
generating training dataset comprising, for each of the plurality of training images, summary statistics of user responses received for the training image;
training, using the training dataset, a machine learning based model configured to receive an input image and predict summary statistics describing a state of mind of a person displayed in the input image;
receiving a new image of a new environment, the new image including a new person;
predicting, using the machine learning based model, summary statistics describing a state of mind of the new person in the new image; and
controlling the moveable device based on the predicted summary statistics describing the state of mind of the new person.
2. The computer-implemented method of claim 1 , wherein sending the plurality of training images to the annotators via the user interface, further comprises:
selecting a set of questions to be presented to the annotators with the plurality of training images; and
sending the set of questions and the plurality of training images for display to the annotators via the user interface.
3. The computer-implemented method of claim 2 , wherein the set of questions are selected based on one or more of: features associated with the moveable device and features extracted a training image.
4. The computer-implemented method of claim 1 , wherein the moveable device is a delivery robot and the state of mind predicted by the machine learning based model indicates a likelihood of the new person performing an action, the action including one of:
interact with the delivery robot to load an object to be delivered,
interact with the delivery robot to pick up an object delivered by the delivery robot, or
move into a path of motion of the delivery robot.
5. The computer-implemented method of claim 4 , wherein responsive to the likelihood of the new person interacting with the delivery robot to load the object or pick up the object being greater than a threshold, controlling the delivery robot to perform one or more of:
requesting credentials from the new person,
staying in place for the new person to approach,
moving toward the new person, or
opening a compartment configured to hold the object.
6. The computer-implemented method of claim 4 , wherein responsive to the likelihood of the new person moving into the path of motion of the delivery robot, controlling the moveable device to perform one or more of:
stopping in place until the new person has passed by the delivery robot,
rerouting to move around the new person,
sounding an alarm, or
moving at a slower speed.
7. The computer-implemented method of claim 1 , wherein the moveable device is a forklift or a factory arm and the state of mind predicted by the machine learning based model indicates a likelihood of the new person being aware of the moveable device.
8. The computer-implemented method of claim 7 , wherein responsive to the likelihood of the new person being aware of the forklift or the factory arm being less than a threshold, controlling the forklift or the factory arm to perform one of:
stop in place, or
sound an alarm.
9. The computer-implemented method of claim 1 , wherein the moveable device is a forklift or a factory arm and wherein the state of mind indicates a likelihood of the new person moving into a path of motion of the forklift or the factory arm.
10. The computer-implemented method of claim 8 , wherein responsive to the likelihood of the new person moving into the path of motion of the moveable device being greater than a threshold, controlling the forklift or the factory arm to perform one of:
stop in place, or
sound an alarm.
11. The computer-implemented method of claim 1 , wherein the moveable device is a drone and the state of mind predicted by the machine learning based model indicates a likelihood of the new person becoming aware of the drone.
12. The computer-implemented method of claim 11 , wherein responsive to the likelihood of the new person becoming aware of the drone being greater than a threshold, controlling the drone to perform one or more of:
sounding an alarm,
flying to a higher altitude, or
moving in a direction away from the new person.
13. The computer-implemented method of claim 11 , wherein the moveable device is a drone and the state of mind predicted by the machine learning based model indicates a likelihood of the new person acting interacting with the drone.
14. The computer-implemented method of claim 13 , wherein responsive to the likelihood of the new person interacting with the drone being greater than a threshold, controlling the drone to perform one or more of:
sounding an alarm,
flying to a higher altitude,
moving in a direction away from the new person, or
contacting a human operator.
15. The computer-implemented method of claim 1 , wherein the moveable device is a robot configured to perform hospitality functions or security functions and the state of mind predicted by the machine learning based model indicates a likelihood of the new person performing an action, the action including one of:
interacting with the robot, or
moving into a path of motion of the robot.
16. The computer-implemented method of claim 15 , wherein responsive to the likelihood of the new person interacting with the robot exceeding a threshold, controlling the robot to move toward the new person and initiate an interaction.
17. The computer-implemented method of claim 15 , wherein responsive to the likelihood of the new person interacting with the robot exceeding a threshold, controlling the robot to perform one or more of:
sounding an alarm,
blocking the new person, or
contacting security personnel.
18. The computer-implemented method of claim 15 , wherein responsive to the likelihood of the new person moving into the path of motion of the robot, controlling the moveable device to perform one or more of:
stopping in place until the new person has passed by the robot,
rerouting to move around the new person,
sounding an alarm, or
moving at a slower speed.
19. A non-transitory computer readable storage medium storing instructions that when executed by a computer processor cause the computer processor to perform steps of a computer-implemented method for controlling a moveable device, the steps comprising:
receiving a plurality of training images, wherein each image from the plurality of training images displays one or more persons;
for each of the plurality of training images:
sending the training image to one or more annotators via a user interface requesting user responses describing a state of mind of a person displayed in the training image; and
receiving user responses from the annotators, each user response describing the state of mind of the one or more persons displayed in the image;
generating training dataset comprising, for each of the plurality of training images, summary statistics of user responses received for the training image;
training, using the training dataset, a machine learning based model configured to receive an input image and predict summary statistics describing a state of mind of a person displayed in the input image;
receiving a new image of a new environment, the new image including a new person;
predicting, using the machine learning based model, summary statistics describing a state of mind of the new person in the new image; and
controlling the moveable device based on the predicted summary statistics describing the state of mind of the new person.
20. A computer system comprising:
a computer processor; and
a non-transitory computer readable storage medium storing instructions that when executed by a computer processor cause the computer processor to perform steps of a computer-implemented method for determining an attribute of one or more persons, the steps comprising:
receiving a plurality of training images, wherein each image from the plurality of training images displays one or more persons;
for each of the plurality of training images:
sending the training image to one or more annotators via a user interface requesting user responses describing a state of mind of a person displayed in the training image; and
receiving user responses from the annotators, each user response describing the state of mind of the one or more persons displayed in the image;
generating training dataset comprising, for each of the plurality of training images, summary statistics of user responses received for the training image;
training, using the training dataset, a machine learning based model configured to receive an input image and predict summary statistics describing a state of mind of a person displayed in the input image;
receiving a new image of a new environment, the new image including a new person;
predicting, using the machine learning based model, summary statistics describing a state of mind of the new person in the new image; and
controlling a moveable device based on the predicted summary statistics describing the state of mind of the new person.