Integration of artificial reality interaction modes
Aspects of the present disclosure are for an interaction mode system that provides multiple interaction modes in an artificial reality environment with automatic, context-specific transitions between interaction modes. An interaction mode can specify how the interaction mode system determines direction indications and movement within an artificial reality environment and interactions for making selections or performing other actions. In some implementations, the interaction mode system can control at least four interaction modes including a no-hands, 3DoF mode, a no-hands, 6DoF mode, a gaze and gesture mode, and a ray casting mode. The interaction mode system can employ a mapping of interaction mode context factors (e.g., which components are enabled, mode settings, lighting or other environment conditions, current body positions, etc.) to interaction modes to control transitioning between particular interaction modes. The interaction mode system can also provide affordances for signaling a current interaction mode and interaction mode transitions.
1. A method for automatically transitioning between interaction modes for interpreting user input in an artificial reality system, the method comprising:
identifying a first interaction mode context indicating that user position tracking input is not available;
in response to identifying the first interaction mode context, enabling a no-hands, three-degree-of-freedom interaction mode;
identifying a second interaction mode context indicating that—
user position tracking input is available, and
hand tracking input is not available or that a first tracked hand pose does not match a hand ready state;
in response to identifying the second interaction mode context, enabling a no-hands, six-degree-of-freedom interaction mode;
identifying a third interaction mode context indicating that a second tracked hand pose matches the hand ready state;
in response to identifying the third interaction mode context, enabling a gaze and gesture interaction mode;
identifying a fourth interaction mode context indicating that a third tracked hand pose matches a ray state; and
in response to identifying the fourth interaction mode context, enabling a ray casting interaction mode.
2. The method of claim 1 , wherein enabling at least one of the interaction modes is further in response to an interaction mode change trigger that:
occurs periodically; or
occurs in response to a change in an interaction mode context identified as a result of monitoring interaction mode context factors.
3. The method of claim 1 further comprising transitioning to an interaction mode based on receiving a user instruction to change from a current interaction mode.
4. The method of claim 1 , wherein the second tracked hand pose that matches the hand ready state comprises a hand pose identified as the user's palm facing at least a threshold amount upward.
5. The method of claim 1 , wherein the third tracked hand pose that matches the ray state comprises a hand pose identified as the user's palm facing at least a threshold amount downward.
6. The method of claim 1 , wherein the no-hands, three-degree-of-freedom interaction mode comprises:
receiving user indications of direction at least based on a determined orientation of the user's head; and
receiving user indications of an action, in relation to one of the user indications of direction, based on a dwell timer;
wherein, while in the no-hands, three-degree-of-freedom interaction mode, user movements in the X, Y, and Z axes do not automatically translate to field-of-view movements in the X, Y, and Z axes in an artificial reality environment.
7. The method of claim 1 , wherein the no-hands, six-degree-of-freedom interaction mode comprises:
receiving user indications of direction at least based on a determined orientation of the user's head;
receiving user indications of an action, in relation to one of the user indications of direction, based on a dwell timer; and
translating user movements in the X, Y, and Z axes to field-of-view movements in the X, Y, and Z axes in an artificial reality environment.
8. The method of claim 1 , wherein the gaze and gesture interaction mode comprises:
receiving user indications of direction at least based on a determined orientation of the user's head and based on a determined position of the user's head in relation to an artificial reality environment, wherein the determined position of the user's head is based on tracked movements of the user along the X, Y, and Z axes; and
receiving user indications of an action, in relation to one of the user indications of direction, by tracking a user's hand postures and matching the hand postures to specified actions.
9. The method of claim 1 , wherein the ray casting interaction mode comprises:
receiving user indications of direction at least based on a ray cast by the artificial reality system, wherein the ray is cast from a location relative to a tracked position of at least one of the user's hands; and
receiving user indications of an action, in relation to the ray, by tracking a user's hand postures and matching the hand postures to specified actions.
10. The method of claim 1 , wherein each of the no-hands, three-degree-of-freedom interaction mode, the no-hands, six-degree-of-freedom interaction mode, and the gaze and gesture interaction mode provides a visual affordance including a gaze cursor, wherein the gaze cursor is illustrated in a field-of-view for the user, positioned based at least in part on a tracked position of the user's head.
11. The method of claim 1 ,
wherein each of the no-hands, three-degree-of-freedom interaction mode, the no-hands, six-degree-of-freedom interaction mode, and the gaze and gesture interaction mode provides a visual affordance including a gaze cursor, and
wherein the gaze cursor is illustrated in a field-of-view for the user, positioned based at least in part on a tracked position of the user's head and based on a tracked gaze direction of the user.
12. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for transitioning between interaction modes for interpreting user input in an artificial reality system, the process comprising:
identifying a first interaction mode context indicating that hand tracking input is not available or that a first tracked hand pose does not match a hand ready state;
in response to identifying the first interaction mode context, enabling a no-hands interaction mode;
identifying a second interaction mode context indicating that a second tracked hand pose matches the hand ready state;
in response to identifying the second interaction mode context, enabling a gaze and gesture interaction mode;
identifying a third interaction mode context indicating that a third tracked hand pose matches a ray state; and
in response to identifying the third interaction mode context, enabling a ray casting interaction mode.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the no-hands interaction mode comprises:
receiving user indications of direction at least based on a determined orientation of the user's head; and
receiving user indications of an action, in relation to one of the user indications of direction, based on a dwell timer;
wherein:
when user position tracking input is not available, the no-hands interaction mode does not translate user movements in the X, Y, and Z axes into field-of-view movements in the X, Y, and Z axes in an artificial reality environment; and
when user position tracking input is available, the no-hands interaction mode automatically translates user movements in the X, Y, and Z axes into field-of-view movements in the X, Y, and Z axes in an artificial reality environment.
14. The non-transitory computer-readable storage medium of claim 12 ,
wherein the gaze and gesture interaction mode provides a visual affordance including a sphere illustrated in a field-of-view of the user positioned between the user's thumb and one other finger; and
wherein the sphere is illustrated as resized or distorted according to a determined distance between the user's thumb and the other finger.
15. The non-transitory computer-readable storage medium of claim 12 ,
wherein the ray casting interaction mode provides a visual affordance including a shape illustrated in a field-of-view of the user positioned between the user's thumb and one other finger; and
wherein the shape is illustrated as resized or distorted according to a determined distance between the user's thumb and the other finger.
16. The non-transitory computer-readable storage medium of claim 12 , wherein the second tracked hand pose that matches the hand ready state comprises a hand pose identified as the user's palm facing upward.
17. The non-transitory computer-readable storage medium of claim 12 , wherein the gaze and gesture interaction mode comprises:
receiving user indications of direction at least based on a determined orientation of the user's head and based on a determined position of the user's head in relation to an artificial reality environment, wherein the determined position of the user's head is based on tracked movements of the user along the X, Y, and Z axes; and
receiving user indications of an action, in relation to one of the user indications of direction, by tracking a user's hand postures and matching the hand postures to specified actions.
18. A computing system for interpreting user input in an artificial reality system, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process for transitioning between interaction modes, the process comprising:
identifying a first interaction mode context indicating that hand tracking input is not available or that a first tracked hand pose does not match a hand ready state;
in response to identifying the first interaction mode context, enabling a no-hands interaction mode;
identifying a second interaction mode context indicating that a second tracked hand pose matches the hand ready state;
in response to identifying the second interaction mode context, enabling a gaze and gesture interaction mode;
identifying a third interaction mode context indicating that a third tracked hand pose matches a ray state; and
in response to identifying the third interaction mode context, enabling a ray casting interaction mode.
19. The computing system of claim 18 , wherein the third tracked hand pose that matches the ray state comprises a hand pose identified as the user's palm facing at least a threshold amount downward.
20. The computing system of claim 18 , wherein the no-hands interaction mode and the gaze and gesture interaction mode provide a visual affordance including a gaze cursor, wherein the gaze cursor is illustrated in a field-of-view for the user, positioned based at least in part on a tracked position of the user's head.