Augmented reality coordination of human-robot interaction
View Patent ↗Systems and methods for human-robot communication. More particularly, some embodiments use augmented reality to facilitate communication of robot intention and teleoperation in human-robot cooperative environments. Various embodiments of the present technology provide a middleware that integrates augmented reality (AR) with novel teleoperation interfaces to increase operation effectiveness, support the user in conducting concurrent work, and decrease stress. Various embodiments provide predictive graphical interfaces such that a teleoperator controls a virtual robot surrogate, rather than directly operating the robot itself, providing the user with foresight regarding where the physical robot will end up and how it will get there. In accordance with various embodiments a user may select between two AR interfaces using such a surrogate: one focused on real-time control and one inspired by waypoint delegation.
1. A method for operating a head-mounted display, the method comprising:
generating an augmented reality visualization of an environment of a user based on:
data collected by a robot collocated with the user wearing a head-mounted display, wherein the data collected from the robot comprises information about a local environment or objects within a past or current environment of the robot; and
a user perspective of the local environment when the user is wearing the head-mounted display; and
displaying the augmented reality visualization of the local environment to the user via an augmented reality or mixed reality view in the head-mounted display.
2. The method of claim 1 , further comprising:
linking the user perspective of the local environment to the data collected by the robot; and
generating the augmented reality visualization of the local environment based on the data linked to the user perspective.
3. The method of claim 2 , wherein the information about the local environment or objects within the past or current environment of the robot comprises one or more of a temperature of the local environment, noise levels in the local environment, radiation levels of the local environment, air quality of the local environment, or chemical composition of the local environment.
4. The method of claim 1 , further comprising:
receiving one or more commands from the user instructing the robot to change robot states;
updating the augmented reality visualization in the head-mounted display to show a result of the one or more commands; and
transmitting the one or more commands to the robot to be executed.
5. The method of claim 1 , wherein generating the augmented reality visualization of the environment of the user comprises:
virtually augmenting the local environment with augmented reality to alter an appearance of the robot or add virtually imagery to local environment of the robot.
6. The method of claim 5 , wherein the robot or the local environment of the robot is virtually augmented to communicate a response of the robot to teleoperation commands and facilitate teleoperation of the robot via a user interface.
7. The method of claim 1 , wherein generating the augmented reality visualization of the environment of the user comprises:
receiving one or more commands from the user instructing the robot to change robot states;
analyzing the one or more commands to identify a hazard that would potentially lead to damage to the robot or the local environment or violate a rule of operation upon execution;
generating a set of one or more modified commands to avoid the hazard;
updating the augmented reality visualization in the head-mounted display to show a result of executing the one or more modified commands by the robot; and
transmit the set of one or more modified commands to the robot to be executed.
8. The method of claim 7 , wherein generating the set of one or more modified commands to avoid the hazard comprises:
determining an intent of the user based on the one or more commands; and
generating the set of one or more modified commands to preserve the intent of the user while avoiding the hazard.
9. The method of claim 7 , wherein the head-mounted display comprises a user interface for teleoperating the robot, and the user interface comprises virtual augmentation comprising one or more of the following:
a real-time virtual surrogate of the robot, such that virtual teleoperation of a real-time virtual surrogate directs teleoperation of the robot;
a waypoint virtual surrogate of the robot, such that manipulation of virtual waypoints using the waypoint virtual surrogate directs teleoperation of the robot; or
one or more non-virtual scene objects of an environmental context in which the robot is acting.
10. The method of claim 1 , wherein the augmented reality visualization comprises a user interface that is virtually augmented to communicate robot motion intent.
11. The method of claim 1 , wherein the augmented reality visualization in the head-mounted display communicates robot motion intent by at least one of the following:
a NavPoints approach;
an Arrows approach;
a Gaze approach; or
a Utilities approach.
12. The method of claim 1 , further comprising tracking where the robot has traveled and wherein generating the augmented reality visualization of the environment of the user comprises:
displaying robot data that identifies where the robot has traveled within the augmented reality visualization of the environment.
13. The method of claim 1 , further comprising generating a datadrop at a location, wherein the datadrop comprises the data collected by a robot.
14. The method of claim 13 , wherein the datadrop is automatically created in response to the robot detecting an event.
15. The method of claim 14 , wherein the event includes one or more sensor parameter crossing a threshold.
16. The method of claim 13 , wherein the datadrop is created in the location in response to a datadrop command from the user.
17. The method of claim 13 , wherein additional robots passing by the location identify the datadrop and any augmented reality or mixed reality view is updated to visualize the data stored within the datadrop.
18. The method of claim 1 , further comprising updating the augmented reality or mixed reality view to include relative position guidelines.
19. The method of claim 1 , wherein the head-mounted display comprises an augmented reality display or a mixed reality display.
20. The method of claim 1 , further comprising:
identifying a position of the robot within the local environment; and
modifying the augmented reality or mixed view in the head-mounted display to include a radar-like design to assist the user in teleoperation of the robot.
21. The method of claim 20 , wherein the robot comprises one of:
a ground-based robot and the position of the robot is on the ground; or
is a flight-capable robot and the position of the robot is in the sky.
22. The method of claim 1 , wherein the data collected from the robot comprises a robot intent or commands issued from the user.
23. A system comprising:
a non-virtual robot comprising sensors or cameras, wherein:
the non-virtual robot collects, using the sensors or the cameras, data as the non-virtual robot navigates an environment; and
the data collected by the non-virtual robot comprises information about a local environment or objects within a past or current environment of the non-virtual robot;
an augmented reality system comprising an augmented reality display, a communications module, a processor, a non-transitory computer readable medium having stored thereon instructions that when executed by the processor cause the augmented reality system to:
generate an augmented reality visualization of the local environment based on a subset of data collected by the non-virtual robot and a user perspective of the local environment when the user is wearing the augmented reality display; and
display the augmented reality visualization of the local environment to the user via the augmented reality display.
24. The system of claim 23 , further comprising a cloud-based platform to store the information about the environment of the non-virtual robot.
25. The system of claim 23 , wherein:
the augmented reality display is associated with one or more sensors to monitor for commands from the user to remotely operate the non-virtual robot; and
the communications module transmits the commands detected by the one or more sensors associated with the augmented reality display and transfers information about one or more robot states of the non-virtual robot to allow the augmented reality visualization to be updated.
26. The system of claim 23 , wherein the augmented reality system comprises one or more interfaces for receiving commands to change a state of the non-virtual robot.
27. The system of claim 26 , wherein the augmented reality system comprises a hazard detection module, under control of the processor, to:
receive one or more commands to change a state of the non-virtual robot;
analyze the one or more commands to identify a hazard that would potentially damage the non-virtual robot, damage an environment of the non-virtual robot, injure a collocated human, or violate an operational rule upon execution; and
generate a set of one or more modified commands to avoid the hazard or violation of the operational rule.
28. The system of claim 27 , wherein the augmented reality system:
update the augmented reality visualization on the augmented reality display to show a result from executing the one or more modified commands by the non-virtual robot; and
transmit, using the communications module, the set of one or more modified commands to the robot to be executed; and
generate imagery via the augmented reality display of a robot state associated with the set of one or more modified commands to provide the user additional insights about the non-virtual robot and a current state of the non-virtual robot.
29. The system of claim 26 , wherein the augmented reality system comprises an intention analyzer configured to:
determine, upon identifying a hazard that would potentially damage the non-virtual robot, damage to an environment of the non-virtual robot, injure a collocated human, or violate an operational rule upon execution, an intent of the user by analyzing the commands; and
generate a set of one or more modified commands to preserve the intent of the user.
30. The system of claim 23 , wherein the augmented reality display comprises a head-mounted display.
31. The system of claim 23 , wherein the non-virtual robot is virtually augmented to facilitate teleoperation of the non-virtual robot using a Callout design.
32. The system of claim 31 , wherein:
the augmented reality display comprises a representation of one or more non-virtual scene objects in an environmental context in which the non-virtual robot is acting; and
the one or more non-virtual scene objects are virtually augmented to communicate robot motion intent.
33. The system of claim 31 , wherein the augmented reality display comprises a representation of virtual imagery added to a representation of the local environment without augmentation of a non-virtual object.
34. The system of claim 31 , wherein:
the augmented reality display comprises a representation of one or more non-virtual scene objects in an environmental context in which the non-virtual robot is actin; and
the representation is virtually augmented to facilitate teleoperation of the non-virtual robot via a user interface or via a Frustum design.
35. The system of claim 31 , wherein:
the sensors of the non-virtual robot comprise non-real-time feedback sensors; and
the augmented reality display comprises a Frustrum design for providing feedback from the non-real-time feedback sensors.
36. The system of claim 35 , wherein the non-real-time feedback sensors comprise at least one of x-ray or lidar.
37. The system of claim 23 , wherein the augmented reality system communicates robot motion intent of the non-virtual robot by at least one of the following:
a NavPoints approach;
an Arrows approach;
a Gaze approach; or
a Utilities approach.
38. A method teleoperating a non-virtual robot the method comprising:
receiving, via an augmented reality system, one or more commands that are likely to cause the non-virtual robot to perform an execution hazard or a rule violation, wherein the execution hazard is predicted to damage the robot, an environment of the non-virtual robot, or a collocated human or result in an operational rule violation;
modifying the one or more commands to avoid the execution hazard or the rule violation; and
generating an augmented reality visualization in the augmented reality system to show a surrogate robot emulating an expected response of the non-virtual robot to the one or more commands as modified to avoid the execution hazard or the rule violation.
39. The method of claim 38 , wherein the rule violation would cause the non-virtual robot to exceed a maximum height, go below a minimum height, exceed a maximum speed, or fall below a minimum speed.
40. The method of claim 38 , further comprising broadcasting to additional augmented reality systems a future state of the non-virtual robot.