Integration of artificial reality interaction modes
Aspects of the present disclosure are for an interaction mode system that provides multiple interaction modes in an artificial reality environment with automatic, context-specific transitions between interaction modes. An interaction mode can specify how the interaction mode system determines direction indications and movement within an artificial reality environment and interactions for making selections or performing other actions. In some implementations, the interaction mode system can control at least four interaction modes including a no-hands, 3DoF mode, a no-hands, 6DoF mode, a gaze and gesture mode, and a ray casting mode. The interaction mode system can employ a mapping of interaction mode context factors (e.g., which components are enabled, mode settings, lighting or other environment conditions, current body positions, etc.) to interaction modes to control transitioning between particular interaction modes. The interaction mode system can also provide affordances for signaling a current interaction mode and interaction mode transitions.
1. A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for transitioning between interaction modes for interpreting user input in an artificial reality system, the process comprising:
identifying a first interaction mode context mapped to a no-hands interaction mode in which at least some user interactions are performed using a gaze cursor;
in response to identifying the first interaction mode context, enabling the no-hands interaction mode, including enabling the gaze cursor;
identifying a second interaction mode context mapped to a gaze and gesture interaction mode in which at least some user interactions are performed through a combination of a tracked user gaze and identified user gestures;
in response to identifying the second interaction mode context, enabling the gaze and gesture interaction mode;
identifying a third interaction mode context mapped to a ray casting interaction mode in which at least some user interactions are performed using a virtual ray projection cast out from a tracked position of a user's hand; and
in response to identifying the third interaction mode context, enabling the ray casting interaction mode.
2. The non-transitory computer-readable storage medium of claim 1 , wherein the no-hands interaction mode comprises:
receiving user indications of direction at least based on a determined orientation of the user's head, wherein a position of the gaze cursor is based on the user indications of direction; and
receiving user indications of an action, in relation to one of the user indications of direction, based on a dwell timer;
wherein:
when user position tracking input is not available, the no-hands interaction mode does not translate user movements in the X, Y, and Z axes into field-of-view movements in the X, Y, and Z axes in an artificial reality environment; and
when user position tracking input is available, the no-hands interaction mode automatically translates user movements in the X, Y, and Z axes into field-of-view movements in the X, Y, and Z axes in an artificial reality environment.
3. The non-transitory computer-readable storage medium of claim 1 , wherein the no-hands interaction mode comprises:
receiving user indications of direction at least based on a determined orientation of the user's head, wherein a position of the gaze cursor is based on the user indications of direction; and
receiving user indications of an action, in relation to one of the user indications of direction, based on a dwell timer.
4. The non-transitory computer-readable storage medium of claim 1 ,
wherein the ray casting interaction mode provides a visual affordance including a shape illustrated in a field-of-view of the user positioned between the user's thumb and one other finger; and
wherein the shape is illustrated as resized or distorted according to a determined distance between the user's thumb and the other finger.
5. The non-transitory computer-readable storage medium of claim 1 ,
wherein the tracked user gaze, of the gaze and gesture interaction mode, is tracked by receiving user indications of direction at least based on a determined orientation of the user's head and based on a determined position of the user's head in relation to an artificial reality environment, wherein the determined position of the user's head is based in part on tracked movements of the user along the X, Y, and Z axes; and
wherein the identified user gestures, of the gaze and gesture interaction mode, are received in relation to one of the user indications of direction and are determined by tracking a user's hand postures and matching at least one of the user's hand postures to specified actions.
6. The non-transitory computer-readable storage medium of claim 1 , wherein the identifying the first interaction mode context includes identifying that hand tracking input is not available or identifying that a first tracked hand pose does not match a hand ready state.
7. The non-transitory computer-readable storage medium of claim 1 , wherein the identifying the second interaction mode context includes identifying that a second tracked hand pose matches a hand ready state that comprises a hand pose identified as the user's palm facing at least a threshold amount upward.
8. The non-transitory computer-readable storage medium of claim 1 , wherein the identifying the third interaction mode context includes identifying that a third tracked hand pose matches a ray state that comprises a hand pose identified as the user's palm facing at least a threshold amount downward.
9. The non-transitory computer-readable storage medium of claim 1 ,
wherein the gaze and gesture interaction mode includes the gaze cursor; and
wherein the gaze cursor is illustrated in a field-of-view for the user, positioned based at least in part on a tracked position of the user's head and based on a tracked gaze direction of the user.
10. The non-transitory computer-readable storage medium of claim 1 ,
wherein the identifying the first interaction mode context includes identifying that user position tracking input is not available;
wherein the enabling the no-hands interaction mode comprises enabling a no-hands, three-degree-of-freedom interaction mode; and
wherein, while in the no-hands, three-degree-of-freedom interaction mode, user movements in the X, Y, and Z axes do not automatically translate to field-of-view movements in the X, Y, and Z axes in an artificial reality environment.
11. The non-transitory computer-readable storage medium of claim 1 ,
wherein the identifying the first interaction mode context includes identifying that:
user position tracking input is available; and
hand tracking input is not available or that a first tracked hand pose does not match a hand ready state;
wherein the enabling the no-hands interaction mode comprises enabling a no-hands, six-degree-of-freedom interaction mode; and
wherein, while in the no-hands, six-degree-of-freedom interaction mode, user movements in the X, Y, and Z axes are automatically translated to field-of-view movements in the X, Y, and Z axes in an artificial reality environment.
12. The non-transitory computer-readable storage medium of claim 1 , wherein the identifying the first interaction mode context, the identifying the second interaction mode context, and/or the identifying the third interaction mode context, is based on receiving a user instruction to change from a current interaction mode to an indicated interaction mode.
13. A computing system for transitioning between interaction modes for interpreting user input in an artificial reality system, the computing system comprising:
one or more processors; and
one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:
identifying a first interaction mode context mapped to a no-hands interaction mode in which at least some user interactions are performed using a gaze cursor;
in response to identifying the first interaction mode context, enabling the no-hands interaction mode, including enabling the gaze cursor;
identifying a second interaction mode context mapped to a gaze and gesture interaction mode in which at least some user interactions are performed through a combination of a tracked user gaze and identified user gestures;
in response to identifying the second interaction mode context, enabling the gaze and gesture interaction mode;
identifying a third interaction mode context mapped to a ray casting interaction mode in which at least some user interactions are performed using a virtual ray projection cast out from a tracked position of a user's hand; and
in response to identifying the third interaction mode context, enabling the ray casting interaction mode.
14. The computing system of claim 13 , wherein the no-hands interaction mode comprises:
receiving user indications of direction at least based on a determined orientation of the user's head, wherein a position of the gaze cursor is based on the user indications of direction; and
receiving user indications of an action, in relation to one of the user indications of direction, based on a dwell timer.
15. The computing system of claim 13 ,
wherein the tracked user gaze, of the gaze and gesture interaction mode, is tracked by receiving user indications of direction at least based on a determined orientation of the user's head and based on a determined position of the user's head in relation to an artificial reality environment, wherein the determined position of the user's head is based in part on tracked movements of the user along the X, Y, and Z axes; and
wherein the identified user gestures, of the gaze and gesture interaction mode, are received in relation to one of the user indications of direction and are determined by tracking a user's hand postures and matching at least one of the user's hand postures to specified actions.
16. The computing system of claim 13 ,
wherein the identifying the second interaction mode context includes identifying that a second tracked hand pose matches a hand ready state that comprises a hand pose identified as the user's palm facing at least a threshold amount upward; or
wherein the identifying the third interaction mode context includes identifying that a third tracked hand pose matches a ray state that comprises a hand pose identified as the user's palm facing at least a threshold amount downward.
17. The computing system of claim 13 , wherein the gaze cursor is illustrated in a field-of-view for the user, positioned based at least in part on a tracked position of the user's head and based on a tracked gaze direction of the user.
18. A method for transitioning between interaction modes for interpreting user input in an artificial reality system, the method comprising:
identifying a first interaction mode context mapped to a no-hands interaction mode in which at least some user interactions are performed using a gaze cursor;
in response to identifying the first interaction mode context, enabling the no-hands interaction mode, including enabling the gaze cursor;
identifying a second interaction mode context mapped to a gaze and gesture interaction mode in which at least some user interactions are performed through a combination of a tracked user gaze and identified user gestures;
in response to identifying the second interaction mode context, enabling the gaze and gesture interaction mode;
identifying a third interaction mode context mapped to a ray casting interaction mode in which at least some user interactions are performed using a virtual ray projection cast out from a tracked position of a user's hand; and
in response to identifying the third interaction mode context, enabling the ray casting interaction mode.
19. The method of claim 18 ,
wherein the tracked user gaze, of the gaze and gesture interaction mode, is tracked by receiving user indications of direction at least based on a determined orientation of the user's head and based on a determined position of the user's head in relation to an artificial reality environment, wherein the determined position of the user's head is based in part on tracked movements of the user along the X, Y, and Z axes; and
wherein the identified user gestures, of the gaze and gesture interaction mode, are received in relation to one of the user indications of direction and are determined by tracking a user's hand postures and matching at least one of the user's hand postures to specified actions.
20. The method of claim 18 ,
wherein the identifying the second interaction mode context includes identifying that a second tracked hand pose matches a hand ready state that comprises a hand pose identified as the user's palm facing at least a threshold amount upward; or
wherein the identifying the third interaction mode context includes identifying that a third tracked hand pose matches a ray state that comprises a hand pose identified as the user's palm facing at least a threshold amount downward.