Mode switching for integrated gestural interaction and multi-user collaboration in immersive virtual reality environments
The technology disclosed relates to user interfaces for controlling augmented reality (AR) or virtual reality (VR) environments. Real and virtual objects can be seamlessly integrated to form an augmented reality by tracking motion of one or more real objects within view of a wearable sensor system. Switching the AR/VR presentation on or off to interact with the real world surrounding them, for example to drink some soda, can be addressed with a convenient mode switching gesture associated with switching between operational modes in a VR/AR enabled device.
1. A method of switching display modes between a first mode in which a live video stream is displayed and a second mode in which a virtual reality video stream is displayed in a virtual reality device including motion tracking by a wearable sensor system, the method including:
detecting by at least one camera from a scene in a real world space proximate to a virtual reality device coupled to and controlled by the wearable sensor system, a user's hand gesture made by a user's hand captured in a video stream by the at least one camera as the user's hand moves within a field of view of the at least one camera and determining from the video stream as captured that the motion of the user's hand captured in the video stream indicates a user's hand gesture indicates to the wearable sensor system to switch display modes of the virtual reality device;
selecting a display mode chosen from a first video stream including live video of the scene in real world space and a second video stream including an output of a virtual reality application; and
responsive to the user's hand gesture detected in the video stream as captured, switching the display modes of the virtual reality device by displaying the video stream selected; whereby the display mode is switched between displaying the live video stream and displaying the virtual reality video stream instead of the live video stream by switching display modes.
2. The method of claim 1 , wherein selecting the display mode further includes toggling between the first video stream and the second video stream.
3. The method of claim 2 , further including switching the display mode back when a second user's hand gesture associated with switching modes is detected.
4. The method of claim 2 , further including switching the display mode back when a reverse user's hand gesture in a direction opposite to a direction of the user's hand gesture associated with switching modes is detected.
5. The method of claim 1 , wherein selecting the display mode further includes selecting the first video stream when a first user's hand gesture is detected and selecting the second video stream when a second user's hand gesture is detected.
6. The method of claim 1 , wherein the user's hand gesture includes a swipe of the user's hand from top to bottom.
7. The method of claim 1 , wherein the user's hand gesture includes a swipe from bottom to top.
8. The method of claim 1 , wherein the user's hand gesture includes a swipe from left to right.
9. The method of claim 1 , wherein the user's hand gesture includes a swipe from right to left.
10. The method of claim 1 , wherein the user's hand gesture includes a spiral.
11. The method of claim 1 , wherein the user's hand gesture includes a swiping motion made in a zone proximate to the virtual reality device.
12. The method of claim 1 , wherein selecting a display mode further includes switching from a video stream including image information gathered using a set of IR pixels and a video stream including image information gathered using a set of RGB pixels.
13. A method of providing an augmented reality environment in which a live video is displayed and a second mode in which a virtual reality video is displayed, the method including:
capturing in images by one or more cameras, motion of a user's hand portion moving at least partially in a field of view of the one or more cameras;
detecting in the images as captured by the one or more cameras, a user's hand gesture indicating a command input to change a mode of operation from a virtual reality mode to a live video display mode or from a live video display mode to a virtual reality mode; and
responsive to the user's hand gesture detected in the images as captured, changing the mode of operation of a wearable display device according to the command input; whereby the wearable display device is switched between displaying live video and displaying virtual reality video instead of the live video by changing the mode of operation.
14. The method of claim 13 , wherein the detecting includes:
sensing a swipe of the user's hand portion in close proximity to one or more cameras.
15. The method of claim 14 , wherein the changing includes:
toggling the wearable display device between the virtual reality mode and the live video display mode responsive to motion of the user's hand portion.
16. The method of claim 13 , further including:
returning the wearable display device to a previous mode upon detecting a second gesture made by the user's hand portion.
17. A non-transitory computer readable storage medium impressed with computer program instructions to switch display modes between a first mode in which a live video stream is displayed and a second mode in which a virtual reality video stream is displayed in a virtual reality device including motion tracking by a wearable sensor system, which instructions, when executed on a processor, implement a method comprising:
detecting by at least one camera from a scene in a real world space proximate to a virtual reality device coupled to and controlled by the wearable sensor system, a user's hand gesture made by a user's hand captured in a video stream by the at least one camera as the user's hand moves within a field of view of the at least one camera and determining from motion of the user's hand captured in the video stream as captured that the motion of the user's hand captured in the video stream indicates a user's hand gesture to the wearable sensor system to switch display modes of the virtual reality device;
selecting a display mode chosen from a first video stream including live video of the scene in real world space and a second video stream including an output of a virtual reality application; and
responsive to the user's hand gesture detected in the video stream as captured, switching display modes of the virtual reality device by displaying the video stream selected; whereby the display mode is switched between displaying a live video stream and displaying a virtual reality video stream instead of the live video stream by switching display modes.
18. A non-transitory computer readable storage medium impressed with computer program instructions to provide an augmented reality environment, the instructions, when executed on a processor, implement a method comprising of:
capturing in images by one or more cameras, motion of a user's hand portion moving at least partially in a field of view of the one or more cameras;
detecting in the images as captured by the one or more cameras, a user's hand gesture is indicating a command input to change a mode of operation from a virtual reality mode to a live video display mode or from a live video display mode to a virtual reality mode; and
responsive to the user's hand gesture detected in the images as captured, changing the mode of operation of a wearable display device according to the command input; whereby the wearable display device is switched between displaying live video and displaying virtual reality video instead of the live video by changing the mode of operation.
19. A system including one or more processors coupled to memory, the memory loaded with computer instructions to switch display modes in a virtual reality device including motion tracking by a wearable sensor system, the instructions, when executed on the processors, implement actions comprising of:
detecting by at least one camera from a scene in a real world space proximate to a virtual reality device coupled to and controlled by the wearable sensor system, a user's hand gesture made by a user's hand captured in a video stream by the at least one camera as the user's hand moves within a field of view of the at least one camera and determining from the video stream as captured that the motion of the user's hand captured in the video stream indicates a user's hand gesture to the wearable sensor system to switch display modes of the virtual reality device;
selecting a display mode chosen from a first video stream including live video of the scene in real world space and a second video stream including an output of a virtual reality application; and
responsive to the user's hand gesture detected in the video stream as captured, switching display modes of the virtual reality device by displaying the video stream selected; whereby the display mode is switched between displaying a live video stream and displaying a virtual reality video stream instead of the live video stream by switching display modes.
20. A system including one or more processors coupled to memory, the memory loaded with computer instructions to provide an augmented reality environment, the instructions, when executed on the processors, implement actions comprising of:
capturing in images by one or more cameras, motion of a user's hand portion moving at least partially in a field of view of the one or more cameras;
detecting in the images as captured by the one or more cameras, a user's hand gesture indicating a command input to change a mode of operation from a virtual reality mode to a live video display mode or from a live video display mode to a virtual reality mode; and
responsive to the user's hand gesture detected in the images as captured, changing the mode of operation of a wearable display device according to the command input; whereby the wearable display device is switched between displaying live video and displaying virtual reality video instead of the live video by changing the mode of operation.