IP Library › Granted Patent US 12663874

Granted Patent B2

US 12663874 · App. 18/884,475 · Granted Jun 23, 2026

Integration of artificial reality interaction modes

Inventor: Hayden Schoen (Evanston, IL)

Assignee: Meta Platforms Technologies, LLC

G06F3/017 G06F3/012 G06F3/013 G06F3/011 G06F3/04815 G06F3/04842

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12663874
App. No.: 18/884,475
Granted: Jun 23, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Aspects of the present disclosure are for an interaction mode system that provides multiple interaction modes in an artificial reality environment with automatic, context-specific transitions between interaction modes. An interaction mode can specify how the interaction mode system determines direction indications and movement within an artificial reality environment and interactions for making selections or performing other actions. In some implementations, the interaction mode system can control at least four interaction modes including a no-hands, 3DoF mode, a no-hands, 6DoF mode, a gaze and gesture mode, and a ray casting mode. The interaction mode system can employ a mapping of interaction mode context factors (e.g., which components are enabled, mode settings, lighting or other environment conditions, current body positions, etc.) to interaction modes to control transitioning between particular interaction modes. The interaction mode system can also provide affordances for signaling a current interaction mode and interaction mode transitions.

Claims (71)

1 . A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system communicatively coupled with an artificial reality system, cause the computing system to perform:

selecting a first interaction mode for interpreting a user input in an artificial reality system by identifying first interaction mode context based on first sensor data obtained by a first set of sensors;

in response to selecting the first interaction mode, enabling the first interaction mode, wherein the first interaction mode includes at least: a no-hands interaction mode in which at least some user interactions are performed using a gaze cursor;

selecting a second interaction mode for interpreting the user input in the artificial reality system by identifying second interaction mode context based on second sensor data obtained by a second set of sensors, wherein the second sensor data is distinct from the first sensor data; and

in response to selecting the second interaction mode, enabling the second interaction mode, wherein the second interaction mode includes at least: gaze and gesture interaction mode in which at least some user interactions are performed through a combination of a tracked user gaze and identified user gestures.

2 . The non-transitory computer-readable storage medium of claim 1 , wherein the first interaction mode comprises:

receiving user indications of direction at least based on a determined orientation of a user's head, wherein a position of the gaze cursor is based on the user indications of direction; and

receiving user indications of an action, in relation to one of the user indications of direction, based on a dwell timer;

wherein:

when user position tracking input is not available, the first interaction mode does not translate user movements in X, Y, and Z axes into field-of-view movements in the X, Y, and Z axes in an artificial reality environment; and

when user position tracking input is available, the first interaction mode automatically translates user movements in the X, Y, and Z axes into field-of-view movements in the X, Y, and Z axes in an artificial reality environment.

3 . The non-transitory computer-readable storage medium of claim 1 , wherein the first interaction mode comprises:

receiving user indications of direction at least based on a determined orientation of a user's head, wherein a position of the gaze cursor is based on the user indications of direction; and

receiving user indications of an action, in relation to one of the user indications of direction, based on a dwell timer.

4 . The non-transitory computer-readable storage medium of claim 1 ,

wherein the tracked user gaze, of the second interaction mode, is tracked by receiving user indications of direction at least based on a determined orientation of a user's head and based on a determined position of the user's head in relation to an artificial reality environment, wherein the determined position of the user's head is based in part on tracked movements of the user along X, Y, and Z axes; and

wherein the identified user gestures, of the second interaction mode, are received in relation to one of the user indications of direction and are determined by tracking a user's hand postures and matching at least one of the user's hand postures to specified actions.

5 . The non-transitory computer-readable storage medium of claim 1 , wherein the identifying the first interaction mode context includes identifying, based on the first sensor data, that hand tracking input is not available or identifying that a first tracked hand pose does not match a hand ready state.

6 . The non-transitory computer-readable storage medium of claim 1 , wherein the identifying the second interaction mode context includes identifying, based on the second sensor data, that a second tracked hand pose matches a hand ready state that comprises a hand pose identified as a user's palm facing at least a threshold amount upward.

7 . The non-transitory computer-readable storage medium of claim 1 ,

wherein the second interaction mode also utilizes the gaze cursor; and

wherein the gaze cursor, in the second interaction mode, is illustrated in a field-of-view for the user, positioned based at least in part on a tracked position of user's head and based on a tracked gaze direction of the user.

8 . The non-transitory computer-readable storage medium of claim 1 ,

wherein the identifying the first interaction mode context includes identifying, based on the first sensor data, that user position tracking input is not available;

wherein the first interaction mode comprises enabling a no-hands, three-degree-of-freedom interaction mode; and

wherein, while in the no-hands, three-degree-of-freedom interaction mode, user movements in X, Y, and Z axes do not automatically translate to field-of-view movements in the X, Y, and Z axes in an artificial reality environment.

9 . The non-transitory computer-readable storage medium of claim 1 ,

wherein the identifying the first interaction mode context, based on the first sensor data, includes identifying that:

user position tracking input is available; and

hand tracking input is not available or that a first tracked hand pose does not match a hand ready state, and

wherein the enabling the first interaction mode comprises enabling a no-hands, six-degree-of-freedom interaction mode; and

wherein, while in the no-hands, six-degree-of-freedom interaction mode, user movements in X, Y, and Z axes are automatically translated to field-of-view movements in the X, Y, and Z axes in an artificial reality environment.

10 . The non-transitory computer-readable storage medium of claim 1 , wherein the instructions, when executed by the computing system, further cause the computing system to perform:

selecting a third interaction mode for interpreting the user input in the artificial reality system by identifying third interaction mode context based on third sensor data obtained by a third set of sensors, wherein the third sensor data is distinct from the first and second sensor data;

in response to selecting the third interaction mode, enabling the third interaction mode, wherein the third interaction mode includes at least: a ray casting interaction mode in which at least some user interactions are performed using a virtual ray projection cast out from a tracked position of a user's hand; and

enabling one of the first interaction mode, the second interaction mode, or the third interaction mode for interpreting the user input in the artificial reality system in response to receiving a user instruction to change from a current interaction mode to an indicated interaction mode including, at least, the first interaction mode, the second interaction mode, and the third interaction mode.

11 . The non-transitory computer-readable storage medium of claim 10 ,

wherein the third interaction mode provides a visual affordance including a shape illustrated in a field-of-view of the user positioned between a user's thumb and at least one other finger; and

wherein the shape is illustrated as resized or distorted according to a determined distance between the user's thumb and the at least one other finger.

12 . The non-transitory computer-readable storage medium of claim 10 , wherein the identifying the third interaction mode context includes identifying, based on the third sensor data, that a third tracked hand pose matches a ray state that comprises a hand pose identified as a user's palm facing at least a threshold amount downward.

13 . A computing system for transitioning between interaction modes for interpreting user input in an artificial reality system, the computing system comprising:

one or more processors; and

one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:

in response to selecting the second interaction mode, enabling the second interaction mode, wherein the second interaction mode includes at least: a gaze and gesture interaction mode in which at least some user interactions are performed through a combination of a tracked user gaze and identified user gestures.

14 . The computing system of claim 13 , wherein the first interaction mode comprises:

receiving user indications of direction at least based on a determined orientation of user's head, wherein a position of the gaze cursor is based on the user indications of direction; and

receiving user indications of an action, in relation to one of the user indications of direction, based on a dwell timer.

15 . The computing system of claim 13 ,

16 . The computing system of claim 13 , wherein the instructions, when executed by the one or more processors, cause the computing system to further perform a process comprising:

17 . The computing system of claim 16 ,

wherein the identifying the second interaction mode context includes identifying, based on the second sensor data, that a second tracked hand pose matches a hand ready state that comprises a hand pose identified as a user's palm facing at least a threshold amount upward; or

wherein the identifying the third interaction mode context includes identifying, based on the third sensor data, that a third tracked hand pose matches a ray state that comprises a hand pose identified as the user's palm facing at least a threshold amount downward.

18 . A method for transitioning between interaction modes for interpreting user input in an artificial reality system, the method comprising:

19 . The method of claim 18 ,

20 . The method of claim 18 ,

wherein the identifying the first interaction mode context includes identifying, based on the first sensor data, that hand tracking input is not available or identifying that a first tracked hand pose does not match a hand ready state and wherein the first interaction mode context is mapped to a no-hands interaction mode.