Tactile interaction in virtual environments
Tactile virtual reality (VR) and/or mixed reality (MR) experiences are described. Techniques described herein include receiving data from a sensor and accessing a position and an orientation of a real object that is physically present in a real scene. Furthermore, techniques described herein include identifying the real object based at least in part on the position and the orientation of the real object and causing a graphical element corresponding to the real object to be rendered on a display of a VR and/or MR display device. The graphical element can be determined based at least in part on a VR and/or MR application. The techniques described herein include determining an interaction with the real object and causing a functionality associated with the graphical element to be performed in the VR or MR environment rendered via the VR and/or MR display device, respectively.
1. A method comprising:
accessing, by one or more processors in a head-mounted display (HMD), video data captured by a camera in the HMD;
identifying, by the one or more processors, based at least in part on the video data, a real object that is physically present in a real scene, the real object being held by a user wearing the HMD;
mapping, by the one or more processors, a virtual object to the real object, the virtual object having a same three-dimensional geometry as the real object, the virtual object being different from the real object;
generating, by the one or more processors, a virtual reality scene based on the video data, the virtual object replacing the real object in the virtual reality scene;
presenting the virtual reality scene in a display of the HMD;
detecting, by the one or more processors, a motion of the real object being held by the user;
determining, by the one or more processors, a functionality of the virtual object based on the motion;
changing, by the one or more processors, a view of the virtual object based on the determined functionality; and
presenting, after changing the view of the virtual object, the virtual reality scene in the display of the HMD.
2. The method as recited in claim 1 , wherein changing the view of the virtual object further includes:
mapping a graphical element to the determined functionality, wherein the virtual object is presented in the virtual reality scene based on the graphical element.
3. The method as recited in claim 1 , wherein presenting the virtual reality scene further includes rendering a graphical element on top of the virtual object.
4. The method as recited in claim 1 , wherein detecting the motion further includes:
detecting a shaking of the real object.
5. The method as recited in claim 4 , further comprising:
causing presentation of virtual rain or virtual snow to fall around the virtual object in the virtual reality scene.
6. The method as recited in claim 1 , wherein detecting the motion further includes:
detecting a throw of the real object.
7. The method as recited in claim 6 , further comprising:
creating a virtual explosion around the virtual object in the virtual reality scene.
8. The method as recited in claim 1 , wherein detecting the motion further includes:
detecting a tap interaction.
9. The method as recited in claim 8 , further comprising:
changing a representation of the virtual object in the virtual reality scene.
10. The method as recited in claim 1 , wherein detecting the motion further includes:
detecting that the real object is placed on a floor.
11. The method as recited in claim 10 , further comprising:
causing additional virtual content to be placed around the virtual object in the virtual reality scene.
12. The method as recited in claim 1 , wherein the real object is a block and the virtual object has a same dimensions and shape as the block.
13. The method as recited in claim 1 , wherein the functionality is associated with a binocular, wherein the binocular enables a user to see Previously Presented virtual content in the virtual reality scene.
14. The method as recited in claim 1 , wherein the functionality is associated with sound, wherein a sound is generated when the real object is moved within a predetermined threshold distance from a ear of the user wearing the HMD.
15. The method as recited in claim 1 , wherein the functionality is associated with a portal.
16. The method as recited in claim 1 , wherein identifying the real object further comprises:
determining a geometry of the real object based on combinations of volumetric data, skeletal data, and perspective data; and
mapping the virtual object to the real object based on the geometry of the real object.
17. A head-mounted display (HMD) comprising:
a display;
a camera for capturing video data in front of the HMD;
a memory comprising instructions; and
one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising:
accessing video data captured by the camera;
identifying, based at least in part on the video data, a real object that is physically present in a real scene, the real object being held by a user wearing the HMD;
mapping a virtual object to the real object, the virtual object having a same three-dimensional geometry as the real object, the virtual object being different from the real object;
generating a virtual reality scene based on the video data, the virtual object replacing the real object in the virtual reality scene;
presenting the virtual reality scene in the display of the HMD;
detecting a motion of the real object being held by the user;
determining a functionality of the virtual object based on the motion;
changing a view of the virtual object based on the determined functionality; and
presenting, after changing the view of the virtual object, the virtual reality scene in the display of the HMD.
18. The HMD as recited in claim 17 , wherein changing the view of the virtual object further includes:
mapping a graphical element to the determined functionality, wherein the virtual object is presented in the virtual reality scene based on the graphical element.
19. The HMD as recited in claim 17 , wherein detecting the motion further includes:
detecting one or more of a shaking of the real object, a throw of the real object, a tap interaction, or placing the real object on a floor.
20. A non-transitory machine-readable storage medium including instructions that, when executed by a processor, cause the processor to perform operations comprising:
accessing video data captured by a camera in in a head-mounted display (HMD);
identifying, based at least in part on the video data, a real object that is physically present in a real scene, the real object being held by a user wearing the HMD;
mapping a virtual object to the real object, the virtual object having a same three-dimensional geometry as the real object, the virtual object being different from the real object;
generating a virtual reality scene based on the video data, the virtual object replacing the real object in the virtual reality scene;
presenting the virtual reality scene in a display of the HMD;
detecting a motion of the real object being held by the user;
determining a functionality of the virtual object based on the motion;
changing a view of the virtual object based on the determined functionality; and
presenting, after changing the view of the virtual object, the virtual reality scene in the display of the HMD.