Robot control, training and collaboration in an immersive virtual reality environment
System and methods to create an immersive virtual environment using a virtual reality system that receives parameters corresponding to a real-world robot. The real-world robot may be simulated to create a virtual robot based on the received parameters. The immersive virtual environment may be transmitted to a user. The user may supply input and interact with the virtual robot. Feedback such as the current state of the virtual robot or the real-world robot may be provided to the user. The user may train the virtual robot. The real-world robot may be programmed based on the virtual robot training.
1. A method for programming a robot, comprising:
creating an immersive virtual environment (IVE) using a virtual reality system (VRS);
receiving, by the VRS, parameters corresponding to a real-world robot;
creating, by the VRS within said IVE, a virtual robot, wherein the virtual robot is a simulation of the real-world robot based on the received parameters;
transmitting, by the VRS, a representation of said IVE to a user;
receiving, by the VRS, input from the user, wherein said VRE is configured to allow the user to interact with the virtual robot using said user input;
providing, by the VRS within said IVE, robot feedback to the user, wherein said robot feedback includes a current state of the virtual robot;
training, in the VRS, the virtual robot in the IVE by the user; and
programming, by the VRS, the real-world robot based on the virtual robot training,
wherein said user interaction includes an egocentric perspective, wherein the user operates the virtual robot from a point of view of the robot, and a robot's end effector directly follows a motion of a hand of the user.
2. The method of claim 1 , wherein an image of an arm of the robot is overlaid where an arm of the user would be in real life.
3. The method of claim 1 , wherein said user interaction includes an exocentric perspective, where the user views the virtual robot inside the IVE from a point of view external to the robot.
4. The method of claim 1 , wherein said user interaction includes controlling the real-world robot in real-time.
5. The method of claim 1 , wherein said IVE includes an augmented environment.
6. The method of claim 1 , wherein said IVE includes a virtual reality environment.
7. The method of claim 1 , wherein the virtual robot is programmed with a series of gestures.
8. The method of claim 1 , wherein the virtual robot is programmed by using a virtual user interface.
9. The method of claim 1 , wherein the virtual robot is programmed with a series of motions.
10. The method of claim 9 , wherein the virtual robot replays the series of motions before the series of motions are programmed into the real-world robot.
11. The method of claim 9 , wherein a plurality of real-world robots are programmed with the series of motions.
12. The method of claim 1 , wherein said robot feedback is supplied via a virtual information display (VID), wherein the virtual information display is one of: a 2D sprite or a 3D mesh, and the VID is configurable to be textured or shaded.
13. The method of claim 12 , wherein the VID is configurable to be at least one of: locked to the virtual robot, locked to a view of the user, or locked to an avatar of the user.
14. The method of claim 1 , wherein the IVE includes mixed reality user interfaces (UIs), wherein the UIs are configured to be at least one of: adjusted by the user, locked to an avatar of the user, locked to the virtual robot, or locked to a view of the user.
15. The method of claim 1 , wherein the user interacts with the IVE using at least one of: a stereoscopic 3D display, a virtual reality headset, or an augmented reality device.
16. The method of claim 1 , wherein the real-world robot includes one of: an industrial robot, a domestic robot, an articulated welding robot, an autonomous robot, a military robot, or a medical robot.
17. The method of claim 1 , wherein the real-world robot is one of: remotely operated a distance from the user, located in a hazardous environment, or dangerous for the user to be in close proximity.
18. The method of claim 1 , wherein said virtual robot controls a plurality of real-world robots.
19. The method of claim 1 , wherein said IVE is configurable to define geometric primitives or detailed 3D mesh objects as overlays of real time sensing data to define geometric regions for interaction for the real-world robot.
20. The method of claim 1 , wherein said IVE is configurable to define virtual constraints for the virtual robot, wherein said virtual constraints are translated physical constraints imposed on the real-world robot.
21. The method of claim 1 , wherein said IVE is configurable to display real-time 2D or 3D video or depth information on a virtual scene overlaid in a geometrically correct manner on any 3D representation in the IVE.
22. The method of claim 1 , wherein said IVE is configurable to display forces applied by the robot, or other objects, as sensed by the real-world robot.
23. The method of claim 1 , wherein said IVE is configurable to display a real time representation of a real life environment with an ability to seamlessly transition between the real life environment and a virtual environment.
24. The method of claim 1 , wherein a plurality of users can interact with the IVE.
25. The method of claim 1 , further comprising:
training said user using IVE on said real-world robot using said virtual robot.
26. The method of claim 25 , wherein said IVE is used as a sandbox for learning robot programming or task plan specification using said virtual robot.