Granted Patent B2

US 12,572,213 · App. 18/220,148 · Granted Mar 10, 2026

Inventors: Jacqueline Fashimpaur (Redmond, WA); Aakar Gupta (Redmond, WA); Tanya Renee Jonker (Seattle, WA)

Assignee: Meta Platforms Technologies, LLC

G06F3/017G06F1/163G06F3/011G06F3/04815G06F3/0485G06T19/006

Abstract

Claims (55)

1 . A computer-implemented method for interacting with virtual objects in an artificial reality environment, the method comprising:

receiving sensed data comprising eye tracking data specifying one or more relative coordinates, in the artificial reality environment, for an input gesture;

identifying a position of a virtual object target of the input gesture based on the one or more relative coordinates;

receiving hand tracking data, defining positions of one or more user hands, specifying the input gesture including a pinch start point;

in accordance with a determination that a first distance between A) the pinch start point and B) a current pinch point is below a threshold, determining that the input gesture is a first type of gesture not including a dragging gesture;

in accordance with a determination that the input gesture is the first type of gesture applying, based on the first distance, the first type of gesture to the virtual object target;

in accordance with a determination that a second distance between C) the pinch start point and D) the current pinch point is above the threshold, determining that the input gesture is a second type of gesture including a dragging gesture; and

in accordance with a determination that the input gesture is the second type of gesture:

simulating inertia for the second type of gesture by calculating a transfer function indicating an amount of continued movement for the second type of gesture following a pinch release point in the hand tracking data, and

applying, after a pinch release point that occurred in the hand tracking data and based on the simulated inertia, the second type of gesture to the virtual object target.

2 . The computer-implemented method of claim 1 , wherein the determining a respective type of gesture for the input gesture comprises mapping E) a determined movement type in the hand tracking data comprising one of: flexion and extension, pronation and supination, or radial and ulnar to F) a corresponding gesture type.

3 . The computer-implemented method of claim 1 , wherein the hand tracking data is received from a wrist mounted sensor.

4 . The computer-implemented method of claim 1 , wherein the hand tracking data is received from a visual sensor of a head-mounted device.

5 . The computer-implemented method of claim 1 , wherein the virtual object target comprises at least one of: a scrollable list, a scrollable object, a virtual area, or a highlighted virtual object.

6 . The computer-implemented method of claim 1 , further comprising:

after determining that the input gesture is the first type of gesture, determining that the input gesture is the second type of gesture by identifying a first motion by a first hand and a second motion by a second hand, wherein the second motion comprises a modification of the input gesture triggered by the first motion.

7 . The computer-implemented method of claim 1 , wherein the virtual object target is a virtual list; and

wherein the applying the second type of gesture to the virtual list based on the simulated inertia includes specifying the continued movement as, absent further interaction, a decreasing speed of scrolling of the virtual list to a defined future stopping time according to the transfer function.

8 . The computer-implemented method of claim 1 , wherein the input gesture is a pinch input gesture.

9 . The computer-implemented method of claim 1 , wherein the identifying the position of the virtual object target of the input gesture based on the one or more relative coordinates includes interpreting the input gesture relative to a previously defined fixed neutral point.

10 . The computer-implemented method of claim 1 , wherein the transfer function is selected from a set of possible transfer functions based on a determination of one or both of previous user scrolling patterns or characteristics of a scrollable list.

11 . A non-transitory computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for interacting with virtual objects in an artificial reality environment, the process comprising:

identifying a position of a virtual object target of an input gesture;

receiving hand tracking data, defining positions of one or more user hands, specifying the input gesture;

in accordance with a determination that a first distance between points at which the input gesture has been performed is below a threshold, determining that the input gesture is a first type of gesture not including a dragging gesture;

in accordance with a determination that the input gesture is the first type of gesture applying, based on the first distance, the first type of gesture to the virtual object target;

in accordance with a determination that a second distance between points at which the input gesture has been performed is above the threshold, determining that the input gesture is a second type of gesture including a dragging gesture; and

in accordance with a determination that the input gesture is the second type of gesture:

simulating inertia for the second type of gesture by calculating a transfer function indicating an amount of continued movement for the second type of gesture following a pinch release point in the hand tracking data, and

applying, after the pinch release point that occurred in the hand tracking data and based on the simulated inertia, the second type of dragging gesture to the virtual object target.

12 . The non-transitory computer-readable storage medium of claim 11 ,

wherein the process further comprises receiving sensed data comprising eye tracking data specifying one or more coordinates, in the artificial reality environment, for the input gesture; and

wherein the identifying the position of the virtual object target of the input gesture is based on the one or more coordinates.

13 . The non-transitory computer-readable storage medium of claim 11 ,

wherein the determining a respective type of gesture for the input gesture comprises mapping A) a determined movement type in the hand tracking data comprising one of:

flexion and extension, pronation and supination, or radial and ulnar to B) a corresponding gesture type.

14 . The non-transitory computer-readable storage medium of claim 11 , wherein the hand tracking data is received from a visual sensor of a head-mounted device.

15 . The non-transitory computer-readable storage medium of claim 11 , wherein the virtual object target comprises at least one of: a scrollable list, a scrollable object, a virtual area, or a highlighted virtual object.

16 . The non-transitory computer-readable storage medium of claim 11 wherein the virtual object target is a virtual list; and

wherein the applying the second type of gesture to the virtual list based on the simulated inertia includes specifying the continued movement as, absent further interaction, a decreasing speed of scrolling of the virtual list to a defined future stopping time according to the transfer function.

17 . A computing system for interacting with virtual objects in an artificial reality environment, the computing system comprising:

one or more processors; and

one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:

receiving sensed data comprising eye tracking data specifying one or more coordinates, in the artificial reality environment, for an input gesture;

identifying a position of a virtual object target of the input gesture based on the one or more coordinates;

receiving hand tracking data, defining positions of one or more user hands, specifying the input gesture;

in accordance with a determination that a first distance between points at which the input gesture has been performed is below a threshold, determining that the input gesture is a first type of gesture not including a dragging gesture;

in accordance with a determination that the input gesture is the first type of gesture applying, based on the first distance, the first type of gesture to the virtual object target;

in accordance with a determination that a second distance between points at which the input gesture has been performed is above the threshold, determining that the input gesture is a second type of gesture including a dragging gesture; and

in accordance with a determination that the input gesture is the second type of gesture, applying, after a pinch release point that occurred in the hand tracking data, the second type gesture to the virtual object target.

18 . The computing system of claim 17 , wherein the identifying the position of the virtual object target of the input gesture based on the one or more coordinates includes interpreting the input gesture relative to a previously defined fixed neutral point.

19 . The computing system of claim 17 ,

wherein the process further comprises simulating inertia for the second type of gesture by calculating a transfer function indicating an amount of continued movement for the second type of gesture following the pinch release point in the hand tracking data; and

wherein the applying the second type of gesture to the virtual object target is based on the simulated inertia.

20 . The computing system of claim 19 , wherein the transfer function is selected from a set of possible transfer functions based on a determination of one or both of previous user scrolling patterns or characteristics of a scrollable list.

Assignments (1)

Continuity (2)

References Cited (400)