Interaction with artificial reality based on physical objects
Disclosed herein are related to a system and a method for controlling a virtual reality based on a physical object. In one aspect, a shape of a hand of a user corresponding to a surface or a structure of a physical object is detected. In one aspect, according to the detected shape of the hand, an interactive feature for the surface or the structure of the physical object is generated in a virtual reality or augmented reality application. In one aspect, a user interaction with the interactive feature is detected. In one aspect, an action of the virtual reality or augmented reality application is initiated, in response to detecting the user interaction with the interactive feature.
1. A system comprising:
a processor configured to:
detect a shape of a hand of a user comprising a grip pose of the hand corresponding to a surface or a structure of a physical object,
select a hand template from a plurality of hand templates according to the detected shape of the hand,
determine a dimension and location of the physical object, according to the hand template,
generate, according to the determined dimension and location of the physical object, an interactive feature for the surface or the structure of the physical object, in a virtual reality or augmented reality application,
detect a user interaction with the interactive feature, and
initiate an action of the virtual reality or augmented reality application, in response to detecting the user interaction with the interactive feature.
2. The system of claim 1 , wherein the processor is configured to:
detect a trajectory of a user finger on the surface of the physical object, and
determine a curvature of the surface of the physical object, according to the detected trajectory, wherein the interactive feature in the virtual reality or augmented reality application is generated according to the determined curvature of the surface of the physical object.
3. The system of claim 1 , further comprising:
a storage configured to store mapping between the plurality of hand templates and corresponding interactive features,
wherein the processor is configured to determine the interactive feature for the surface or the structure of the physical object according to the mapping.
4. The system of claim 1 , wherein the processor is configured to generate, according to the detected shape of the hand, the interactive feature for the surface or the structure of the physical object, in the virtual reality or augmented reality application by:
detecting a predetermined interaction of the user associated with the physical object, and
activating the interactive feature, in response to detecting the predetermined interaction associated with the physical object.
5. The system of claim 4 , wherein the predetermined interaction includes a drawing motion of a mark using another hand of the user, relative to the surface of the physical object.
6. The system of claim 5 , wherein the processor is configured to activate the interactive feature by:
generating an image of a virtual button corresponding to the surface or the structure of the physical object, in response to detecting the drawing motion of the mark, and
overlaying the image of the virtual button on the surface or the structure of the physical object.
7. The system of claim 4 , wherein the predetermined interaction comprises a tap by a finger of the hand on the surface of the physical object.
8. The system of claim 7 , wherein the processor is configured to detect the user interaction with the interactive feature by:
tracking a position of the finger of the hand with respect to a location of the tap on the surface of the physical object.
9. The system of claim 4 , wherein the predetermined interaction comprises pulling a trigger of the physical object a specific number of times within a time period.
10. The system of claim 9 , wherein the processor is configured to detect the user interaction with the interactive feature by:
detecting whether the hand performs a motion to pull the trigger of the physical object.
11. A method comprising:
detecting a shape of a hand of a user comprising a grip pose of the hand corresponding to a surface or a structure of a physical object;
selecting a hand template of a plurality of hand templates according to the detected shape of the hand;
determining a dimension and location of the physical object, according to the hand template;
generating, according to the determined dimension and location of the physical object, an interactive feature for the surface or the structure of the physical object, in a virtual reality or augmented reality application;
detecting a user interaction with the interactive feature; and
initiating an action of the virtual reality or augmented reality application, in response to detecting the user interaction with the interactive feature.
12. The method of claim 11 , further comprising:
detecting a trajectory of a user finger on the surface of the physical object, and
determining a curvature of the surface of the physical object, according to the detected trajectory, wherein the interactive feature in the virtual reality or augmented reality application is generated based on the determined curvature of the surface of the physical object.
13. The method of claim 11 , further comprising:
storing mapping between the plurality of hand templates and corresponding interactive features,
wherein the interactive feature for the surface or the structure of the physical object is determined according to the mapping.
14. The method of claim 11 , wherein generating, according to the detected shape of the hand, the interactive feature for the surface or the structure of the physical object, in the virtual reality or augmented reality application includes:
detecting a predetermined interaction of the user associated with the physical object, and
activating the interactive feature, in response to detecting the predetermined interaction associated with the physical object.
15. The method of claim 14 , wherein the predetermined interaction is a drawing motion of a mark using another hand of the user, relative to the surface of the physical object.
16. The method of claim 15 , wherein activating the interactive feature includes:
generating an image of a virtual button corresponding to the surface or the structure of the physical object, in response to detecting the drawing motion of the mark using the another hand of the user, and
overlaying the image of the virtual button on the surface or the structure of the physical object.
17. The method of claim 14 , wherein the predetermined interaction comprises a tap by a finger of the hand on the surface of the physical object.
18. The method of claim 17 , wherein detecting the user interaction with the interactive feature includes:
tracking a position of the finger of the hand with respect to a location of the tap on the surface of the physical object.
19. The method of claim 14 , wherein the predetermined interaction includes pulling a trigger of the physical object a specific number of times within a time period.
20. The method of claim 19 , wherein detecting the user interaction with the interactive feature includes:
detecting whether the hand performs a motion to pull the trigger of the physical object.