Global and local mode virtual object interactions
The present technology relates to artificial reality systems. Such systems provide projections a user can create to specify object interactions. For example, when a user wishes to interact with an object outside her immediate reach, she can use a projection to select, move, or otherwise interact with the distant object. The present technology also includes object selection techniques for identifying and disambiguating between objects, allowing a user to select objects both near and distant from the user. Yet further aspects of the present technology include techniques for interpreting various bimanual (two-handed) gestures for interacting with objects. The present technology further includes a model for differentiating between global and local modes for, e.g., providing different input modalities or interpretations of user gestures.
1. A method for interpreting interactions with objects in an artificial reality environment, the method comprising:
identifying a trigger corresponding to a switch between two or more operational modes,
wherein identifying the trigger includes:
identifying a hand orientation that is mapped to switching between a global mode for manipulation of a target object in spatial relation to the artificial reality environment and a local mode for interacting with interfaces of the target object; or
identifying that a tracked gaze of a user has lingered on the target object for a threshold amount of time which corresponds to switching between the global mode for manipulation of a target object in spatial relation to the artificial reality environment and the local mode for interacting with interfaces of the target object; and
wherein the trigger causes a change from the global mode to the local mode;
receiving a user input as a particular hand gesture in relation to the target object, wherein the particular hand gesture includes a hand pose and motion; and
interpreting, based on the trigger causing the switching to the local mode for interacting with interfaces of the target object, the hand pose and motion of the particular hand gesture as an activation of an interface of the target object, wherein the hand pose and motion of the particular hand gesture that is interpreted as an activation of an interface of the target object also corresponds to a manipulation of the target object in spatial relation to the artificial reality environment when in the global mode.
2. The method of claim 1 , wherein:
the trigger is a first trigger; and
the method further comprises:
identifying a second trigger that causes a change from the local mode to the global mode;
while in the global mode, receiving the particular hand gesture in relation to the target object; and
interpreting, based on the global mode, the hand pose and motion of the particular hand gesture as the manipulation of the target object in spatial relation to the artificial reality environment, wherein the particular hand gesture corresponds to the activation of the interface of the target object when in the local mode.
3. The method of claim 2 , wherein:
the global mode uses casting of projection types specified for use in global mode.
4. The method of claim 2 wherein, the interpreting, based on the global mode, the hand pose and motion of the particular hand gesture as the manipulation of the target object in spatial relation to the artificial reality environment comprises:
selecting, based on the second trigger, a mapping of gestures to actions in the global mode; and
using the mapping to select the manipulation of the target object in spatial relation to the artificial reality environment.
5. The method of claim 1 wherein, the interpreting, based on the local mode, the hand pose and motion of the particular hand gesture as the activation of the interface of the target object comprises:
selecting, based on the trigger, a mapping of gestures to actions in the local mode; and
using the mapping to select the activation of the interface of the target object.
6. The method of claim 1 , wherein identifying the trigger includes identifying the hand orientation, that is mapped to switching between the global and the local modes.
7. The method of claim 1 , wherein identifying the trigger includes identifying that the tracked gaze of the user has lingered on the target object for the threshold amount of time.
8. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform operations for selecting an input modality for interacting with objects in an artificial reality environment, the operations comprising:
identifying a trigger corresponding to a switch between two or more operational modes,
wherein identifying the trigger includes:
identifying a hand orientation that is mapped to switching between a global mode for manipulation of a target object in spatial relation to the artificial reality environment and a local mode for interacting with interfaces of the target object; or
identifying that a tracked gaze of a user has lingered on the target object for a threshold amount of time which corresponds to switching between the global mode for manipulation of a target object in spatial relation to the artificial reality environment and the local mode for interacting with interfaces of the target object; and
wherein the trigger causes a change from the global mode to the local mode;
receiving a user input as a particular hand gesture in relation to the target object, wherein the particular hand gesture includes a hand pose and motion; and
interpreting, based on the trigger causing the switching to the local mode for interacting with interfaces of the target object, the hand pose and motion of the particular hand gesture as an activation of an interface of the target object, wherein the hand pose and motion of the particular hand gesture that is interpreted as an activation of an interface of the target object also corresponds to a manipulation of the target object in spatial relation to the artificial reality environment when in the global mode.
9. The computer-readable storage medium of claim 8 , wherein identifying the trigger includes identifying the hand orientation that is mapped to switching between global and local modes.
10. The computer-readable storage medium of claim 8 wherein,
identifying the trigger includes identifying that the tracked gaze of the user has lingered on the target object for the threshold amount of time.
11. The computer-readable storage medium of claim 8 , wherein the local mode uses a mapping of movements of individual fingers to actuations of interfaces of the target object.
12. The computer-readable storage medium of claim 8 , wherein the global mode uses ray projection casting.
13. The computer-readable storage medium of claim 8 , wherein the global mode uses a projection casting input modality of a cylinder projection type, a sphere projection type, or a cone projection type.
14. A computing system comprising:
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising:
identifying a trigger corresponding to a switch between two or more operational modes, wherein identifying the trigger includes:
identifying a hand orientation that is mapped to switching between a global mode for manipulation of a target object in spatial relation to the artificial reality environment and a local mode for interacting with interfaces of the target object; or
identifying that a tracked gaze of a user has lingered on the target object for a threshold amount of time which corresponds to switching between the global mode for manipulation of a target object in spatial relation to the artificial reality environment and the local mode for interacting with interfaces of the target object; and
in response to the trigger, causing a change from the local mode to the global mode; and
in response to the global mode being active, interpreting a hand pose and motion of a hand gesture by mapping the hand gesture to a first action in the first set of actions, wherein the hand gesture includes the hand pose and motion, and the hand gesture is mapped to a second action in the second set of actions, and wherein the second set of actions is selected when then local mode is active.
15. The computing system of claim 14 , wherein identifying the trigger comprises identifying the hand orientation mapped to switching between global and local modes.
16. The computing system of claim 14 , wherein identifying the trigger comprises determining the tracked gaze of the user has lingered on the target object for the threshold amount of time.
17. The computing system of claim 14 ,
wherein the trigger is a first trigger and the threshold amount of time is a first threshold amount of time; and
wherein the operations further comprise identifying a second trigger to switch from the global mode to the local mode by identifying that the tracked gaze of the user has moved away from the target object for a second threshold amount of time.
18. The computing system of claim 14 , wherein interpreting the hand pose and motion of the hand gesture comprises interpreting the hand pose and motion of the hand gesture as casting a projection with a projection type of a ray projection, a cylinder projection, a sphere projection, a cone projection, or any combination thereof.