Granted Patent B2

US 12,663,874 · App. 18/884,475 · Granted Jun 23, 2026

Inventor: Hayden Schoen (Evanston, IL)

Assignee: Meta Platforms Technologies, LLC

G06F3/017G06F3/012G06F3/013G06F3/011G06F3/04815G06F3/04842

Abstract

Claims (71)

1 . A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system communicatively coupled with an artificial reality system, cause the computing system to perform:

selecting a first interaction mode for interpreting a user input in an artificial reality system by identifying first interaction mode context based on first sensor data obtained by a first set of sensors;

in response to selecting the first interaction mode, enabling the first interaction mode, wherein the first interaction mode includes at least: a no-hands interaction mode in which at least some user interactions are performed using a gaze cursor;

selecting a second interaction mode for interpreting the user input in the artificial reality system by identifying second interaction mode context based on second sensor data obtained by a second set of sensors, wherein the second sensor data is distinct from the first sensor data; and

in response to selecting the second interaction mode, enabling the second interaction mode, wherein the second interaction mode includes at least: gaze and gesture interaction mode in which at least some user interactions are performed through a combination of a tracked user gaze and identified user gestures.

2 . The non-transitory computer-readable storage medium of claim 1 , wherein the first interaction mode comprises:

receiving user indications of direction at least based on a determined orientation of a user's head, wherein a position of the gaze cursor is based on the user indications of direction; and

receiving user indications of an action, in relation to one of the user indications of direction, based on a dwell timer;

wherein:

when user position tracking input is not available, the first interaction mode does not translate user movements in X, Y, and Z axes into field-of-view movements in the X, Y, and Z axes in an artificial reality environment; and

when user position tracking input is available, the first interaction mode automatically translates user movements in the X, Y, and Z axes into field-of-view movements in the X, Y, and Z axes in an artificial reality environment.

3 . The non-transitory computer-readable storage medium of claim 1 , wherein the first interaction mode comprises:

receiving user indications of direction at least based on a determined orientation of a user's head, wherein a position of the gaze cursor is based on the user indications of direction; and

receiving user indications of an action, in relation to one of the user indications of direction, based on a dwell timer.

4 . The non-transitory computer-readable storage medium of claim 1 ,

wherein the tracked user gaze, of the second interaction mode, is tracked by receiving user indications of direction at least based on a determined orientation of a user's head and based on a determined position of the user's head in relation to an artificial reality environment, wherein the determined position of the user's head is based in part on tracked movements of the user along X, Y, and Z axes; and

wherein the identified user gestures, of the second interaction mode, are received in relation to one of the user indications of direction and are determined by tracking a user's hand postures and matching at least one of the user's hand postures to specified actions.

5 . The non-transitory computer-readable storage medium of claim 1 , wherein the identifying the first interaction mode context includes identifying, based on the first sensor data, that hand tracking input is not available or identifying that a first tracked hand pose does not match a hand ready state.

6 . The non-transitory computer-readable storage medium of claim 1 , wherein the identifying the second interaction mode context includes identifying, based on the second sensor data, that a second tracked hand pose matches a hand ready state that comprises a hand pose identified as a user's palm facing at least a threshold amount upward.

7 . The non-transitory computer-readable storage medium of claim 1 ,

wherein the second interaction mode also utilizes the gaze cursor; and

wherein the gaze cursor, in the second interaction mode, is illustrated in a field-of-view for the user, positioned based at least in part on a tracked position of user's head and based on a tracked gaze direction of the user.

8 . The non-transitory computer-readable storage medium of claim 1 ,

wherein the identifying the first interaction mode context includes identifying, based on the first sensor data, that user position tracking input is not available;

wherein the first interaction mode comprises enabling a no-hands, three-degree-of-freedom interaction mode; and

wherein, while in the no-hands, three-degree-of-freedom interaction mode, user movements in X, Y, and Z axes do not automatically translate to field-of-view movements in the X, Y, and Z axes in an artificial reality environment.

9 . The non-transitory computer-readable storage medium of claim 1 ,

wherein the identifying the first interaction mode context, based on the first sensor data, includes identifying that:

user position tracking input is available; and

hand tracking input is not available or that a first tracked hand pose does not match a hand ready state, and

wherein the enabling the first interaction mode comprises enabling a no-hands, six-degree-of-freedom interaction mode; and

wherein, while in the no-hands, six-degree-of-freedom interaction mode, user movements in X, Y, and Z axes are automatically translated to field-of-view movements in the X, Y, and Z axes in an artificial reality environment.

10 . The non-transitory computer-readable storage medium of claim 1 , wherein the instructions, when executed by the computing system, further cause the computing system to perform:

selecting a third interaction mode for interpreting the user input in the artificial reality system by identifying third interaction mode context based on third sensor data obtained by a third set of sensors, wherein the third sensor data is distinct from the first and second sensor data;

in response to selecting the third interaction mode, enabling the third interaction mode, wherein the third interaction mode includes at least: a ray casting interaction mode in which at least some user interactions are performed using a virtual ray projection cast out from a tracked position of a user's hand; and

enabling one of the first interaction mode, the second interaction mode, or the third interaction mode for interpreting the user input in the artificial reality system in response to receiving a user instruction to change from a current interaction mode to an indicated interaction mode including, at least, the first interaction mode, the second interaction mode, and the third interaction mode.

11 . The non-transitory computer-readable storage medium of claim 10 ,

wherein the third interaction mode provides a visual affordance including a shape illustrated in a field-of-view of the user positioned between a user's thumb and at least one other finger; and

wherein the shape is illustrated as resized or distorted according to a determined distance between the user's thumb and the at least one other finger.

12 . The non-transitory computer-readable storage medium of claim 10 , wherein the identifying the third interaction mode context includes identifying, based on the third sensor data, that a third tracked hand pose matches a ray state that comprises a hand pose identified as a user's palm facing at least a threshold amount downward.

13 . A computing system for transitioning between interaction modes for interpreting user input in an artificial reality system, the computing system comprising:

one or more processors; and

one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:

selecting a first interaction mode for interpreting a user input in an artificial reality system by identifying first interaction mode context based on first sensor data obtained by a first set of sensors;

in response to selecting the first interaction mode, enabling the first interaction mode, wherein the first interaction mode includes at least: a no-hands interaction mode in which at least some user interactions are performed using a gaze cursor;

selecting a second interaction mode for interpreting the user input in the artificial reality system by identifying second interaction mode context based on second sensor data obtained by a second set of sensors, wherein the second sensor data is distinct from the first sensor data; and

in response to selecting the second interaction mode, enabling the second interaction mode, wherein the second interaction mode includes at least: a gaze and gesture interaction mode in which at least some user interactions are performed through a combination of a tracked user gaze and identified user gestures.

14 . The computing system of claim 13 , wherein the first interaction mode comprises:

receiving user indications of direction at least based on a determined orientation of user's head, wherein a position of the gaze cursor is based on the user indications of direction; and

receiving user indications of an action, in relation to one of the user indications of direction, based on a dwell timer.

15 . The computing system of claim 13 ,

wherein the tracked user gaze, of the second interaction mode, is tracked by receiving user indications of direction at least based on a determined orientation of a user's head and based on a determined position of the user's head in relation to an artificial reality environment, wherein the determined position of the user's head is based in part on tracked movements of the user along X, Y, and Z axes; and

wherein the identified user gestures, of the second interaction mode, are received in relation to one of the user indications of direction and are determined by tracking a user's hand postures and matching at least one of the user's hand postures to specified actions.

16 . The computing system of claim 13 , wherein the instructions, when executed by the one or more processors, cause the computing system to further perform a process comprising:

selecting a third interaction mode for interpreting the user input in the artificial reality system by identifying third interaction mode context based on third sensor data obtained by a third set of sensors, wherein the third sensor data is distinct from the first and second sensor data;

in response to selecting the third interaction mode, enabling the third interaction mode, wherein the third interaction mode includes at least: a ray casting interaction mode in which at least some user interactions are performed using a virtual ray projection cast out from a tracked position of a user's hand; and

enabling one of the first interaction mode, the second interaction mode, or the third interaction mode for interpreting the user input in the artificial reality system in response to receiving a user instruction to change from a current interaction mode to an indicated interaction mode including, at least, the first interaction mode, the second interaction mode, and the third interaction mode.

17 . The computing system of claim 16 ,

wherein the identifying the second interaction mode context includes identifying, based on the second sensor data, that a second tracked hand pose matches a hand ready state that comprises a hand pose identified as a user's palm facing at least a threshold amount upward; or

wherein the identifying the third interaction mode context includes identifying, based on the third sensor data, that a third tracked hand pose matches a ray state that comprises a hand pose identified as the user's palm facing at least a threshold amount downward.

18 . A method for transitioning between interaction modes for interpreting user input in an artificial reality system, the method comprising:

selecting a first interaction mode for interpreting a user input in an artificial reality system by identifying first interaction mode context based on first sensor data obtained by a first set of sensors;

in response to selecting the first interaction mode, enabling the first interaction mode, wherein the first interaction mode includes at least: a no-hands interaction mode in which at least some user interactions are performed using a gaze cursor;

selecting a second interaction mode for interpreting the user input in the artificial reality system by identifying second interaction mode context based on second sensor data obtained by a second set of sensors, wherein the second sensor data is distinct from the first data; and

in response to selecting the second interaction mode, enabling the second interaction mode, wherein the second interaction mode includes at least: a gaze and gesture interaction mode in which at least some user interactions are performed through a combination of a tracked user gaze and identified user gestures.

19 . The method of claim 18 ,

wherein the tracked user gaze, of the second interaction mode, is tracked by receiving user indications of direction at least based on a determined orientation of a user's head and based on a determined position of the user's head in relation to an artificial reality environment, wherein the determined position of the user's head is based in part on tracked movements of the user along X, Y, and Z axes; and

wherein the identified user gestures, of the second interaction mode, are received in relation to one of the user indications of direction and are determined by tracking a user's hand postures and matching at least one of the user's hand postures to specified actions.

20 . The method of claim 18 ,

wherein the identifying the second interaction mode context includes identifying, based on the second sensor data, that a second tracked hand pose matches a hand ready state that comprises a hand pose identified as a user's palm facing at least a threshold amount upward and wherein the second interaction mode context is mapped to a gaze and gesture interaction mode; or

wherein the identifying the first interaction mode context includes identifying, based on the first sensor data, that hand tracking input is not available or identifying that a first tracked hand pose does not match a hand ready state and wherein the first interaction mode context is mapped to a no-hands interaction mode.

Assignments (2)

Continuity (4)

References Cited (317)