Mapped direct touch virtual trackpad and invisible mouse
A virtual trackpad engine is configured to perform a technique for accessing and manipulating distant virtual objects in an XR environment. The technique includes generating, in an extended reality (XR) environment, a virtual model corresponding to a hand of a user. The technique also includes generating a virtual trackpad in the XR environment, where a surface of the virtual trackpad is mapped to a surface of a distant virtual object in the XR environment. Further, the technique includes determining if the virtual trackpad is activated and responsive to a determination that the virtual trackpad is activated, tracking a movement of the virtual hand. The technique further includes determining an action to be performed in relation to the distant virtual object that corresponds to the movement and, subsequently, performing the action.
1. A computer-implemented method, comprising:
obtaining, in an extended reality (XR) environment, a virtual model corresponding to a hand of a user in a physical environment and used to represent a location of a virtual representation of the hand of the user within the XR environment;
identifying a virtual trackpad in the XR environment, wherein:
a surface of the virtual trackpad is configured to control a focus selector mapped to a surface of a distant virtual object in the XR environment,
the virtual trackpad is presented at a first location distinct from the location of the virtual representation of the hand of the user, and
the virtual trackpad is displayed in the XR environment generated within a near-eye display (NED) system;
in accordance with a determination that the virtual representation of the hand of the user is within a proximity threshold to the first location of the virtual trackpad, activating the virtual trackpad;
responsive to a determination that the virtual trackpad is activated, tracking a movement of the virtual representation of the hand and at least one finger associated with the virtual representation of the hand of the user, such that a sensitivity using to track the at least one finger is distinct from a different sensitivity used to track the virtual representation of the hand of the user;
responsive to a determination that the virtual trackpad is activated, tracking a movement of the virtual representation of the hand and at least one finger associated with the virtual representation of the hand of the user using the sensitivity;
determining an action to be performed in relation to the distant virtual object that corresponds to the movement; and
performing the action.
2. The computer-implemented method of claim 1 , wherein the surface of the virtual trackpad is mapped in its entirety to the surface of the distant virtual object, and wherein each area of the virtual trackpad correlates to a specific location on the distant virtual object.
3. The computer-implemented method of claim 1 , wherein the surface of the virtual trackpad is mapped to the surface of the distant virtual object by utilizing a gain function, and wherein the gain function maps a relative movement of a finger associated with the virtual model to a corresponding action associated with the distant virtual object.
4. The computer-implemented method of claim 3 , wherein the relative movement is a movement of a thumb associated with the virtual model and the corresponding action is a movement of a cursor on the surface of the distant virtual object.
5. The computer-implemented method of claim 1 , wherein the movement is a movement of an index finger associated with the virtual model and the action is a selection of an item on the surface of the distant virtual object.
6. The computer-implemented method of claim 1 , wherein the movement is a movement of an index finger associated with the virtual model and the action is a movement of a cursor on the surface of the distant virtual object.
7. The computer-implemented method of claim 1 , wherein determining if the virtual trackpad is activated comprises determining if a thumb associated with the virtual model is within the proximity threshold of the virtual trackpad.
8. The computer-implemented method of claim 1 further comprising: mapping the virtual trackpad to a real-world object.
9. The computer-implemented method of claim 1 , wherein sensitivity settings for tracking the movement of the virtual representation of the hand are user-configurable.
10. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
obtaining, in an extended reality (XR) environment, a virtual model corresponding to a hand of a user in a physical environment and used to represent a location of a virtual representation of the hand of the user within the XR environment;
identifying a virtual trackpad in the XR environment, wherein:
a surface of the virtual trackpad is configured to control a focus selector mapped to a surface of a distant virtual object in the XR environment,
the virtual trackpad is presented at a first location distinct from the location of the virtual representation of the hand of the user, and
the virtual trackpad is displayed in the XR environment generated within an NED system;
in accordance with a determination that the virtual representation of the hand of the user is within a proximity threshold to the first location of the virtual trackpad, activating the virtual trackpad;
responsive to a determination that the virtual trackpad is activated, tracking a movement of the virtual representation of the hand and at least one finger associated with the virtual representation of the hand of the user, such that a sensitivity using to track the at least one finger is distinct from a different sensitivity used to track the virtual representation of the hand of the user;
responsive to a determination that the virtual trackpad is activated, tracking a movement of the virtual representation of the hand and at least one finger associated with the virtual representation of the hand of the user using the sensitivity;
determining an action to be performed in relation to the distant virtual object that corresponds to the movement; and
performing the action.
11. The one or more non-transitory computer-readable media of claim 10 , wherein the surface of the virtual trackpad is mapped in its entirety to the surface of the distant virtual object, and wherein each area of the virtual trackpad correlates to a specific location on the distant virtual object.
12. The one or more non-transitory computer-readable media of claim 10 , wherein the surface of the virtual trackpad is mapped to the surface of the distant virtual object by utilizing a gain function, and wherein the gain function maps a relative movement of a finger associated with the virtual model to a corresponding action associated with the distant virtual object.
13. The one or more non-transitory computer-readable media of claim 12 , wherein the relative movement is a movement of a thumb associated with the virtual model and the corresponding action is a movement of a cursor on the surface of the distant virtual object.
14. The one or more non-transitory computer-readable media of claim 10 , wherein the movement is a movement of an index finger associated with the virtual model and the action is a selection of an item on the surface of the distant virtual object.
15. The one or more non-transitory computer-readable media of claim 10 , wherein the movement is a movement of an index finger associated with the virtual model and the action is a movement of a cursor on the surface of the distant virtual object.
16. The one or more non-transitory computer-readable media of claim 10 , wherein determining if the virtual trackpad is activated comprises determining if a thumb associated with the virtual model is within the proximity threshold of the virtual trackpad.
17. A wearable device, comprising:
one or more memories that store instructions, and
one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to:
obtain, in an extended reality (XR) environment, a virtual model corresponding to a hand of a user in a physical environment and used to represent a location of a virtual representation of the hand of the user within the XR environment;
identify a virtual trackpad in the XR environment, wherein:
a surface of the virtual trackpad is configured to control a focus selector mapped to a surface of a distant virtual object in the XR environment,
the virtual trackpad is presented at a first location distinct from the location of the virtual representation of the hand of the user, and
the virtual trackpad is displayed in the XR environment generated within an NED system;
in accordance with a determination that the virtual representation of the hand of the user is within a proximity threshold to the first location of the virtual trackpad, activate the virtual trackpad;
responsive to a determination that the virtual trackpad is activated, track a movement of the virtual representation of the hand and at least one finger associated with the virtual representation of the hand of the user, such that a sensitivity using to track the at least one finger is distinct from a different sensitivity used to track the virtual representation of the hand of the user;
responsive to a determination that the virtual trackpad is activated, track a movement of the virtual representation of the hand and at least one finger associated with the virtual representation of the hand of the user using the sensitivity;
determine an action to be performed in relation to the distant virtual object that corresponds to the movement; and
perform the action.
18. The wearable device of claim 17 , wherein sensitivity settings for tracking the movement of the virtual representation of the hand are user-configurable.