Granted Patent B1

US 12,619,316 · App. 18/068,539 · Granted May 5, 2026

Inventors: Annika Rodrigues (Lynnwood, WA); Jenna Velez (Seattle, WA)

Assignee: Meta Platforms Technologies, LLC

G06F3/017G06F3/013G06T19/003H04L51/04H04M1/72433G06T2219/024

Abstract

Claims (75)

1 . A method for an artificial reality system to respond to gestural input, the method comprising:

identifying an event trigger comprising identifying an indication of a received message;

receiving, from a gesture-tracking system, a pinch gesture in relation to the event trigger, wherein the pinch gesture is received in response to the gesture-tracking system detecting that a tip of a thumb of a user and a tip of one other finger of the user touching each other;

determining a first action, of multiple actions associated with the pinch gesture, based on a mapping of A) a context determined when the pinch gesture was made to B) the first action of the multiple actions associated with the pinch gesture, wherein the context includes one or more of a discernable state of the user, surroundings of the user, and one or more recent actions of the user relative to when the pinch gesture was made;

in response to the pinch gesture, taking the first action mapped to the pinch gesture in the context, the first action comprising recording a response to the received message;

receiving, from the gesture-tracking system, a thumb-swipe or a snap gesture, wherein the thumb-swipe or the snap gesture is received in response to the gesture-tracking system detecting either A) that the user has moved, above a threshold speed, the tip of the thumb away and to the side from the tip of the one other finger or B) that the user has moved the tip of the thumb away from the tip of the one other finger and along a side of the one other finger; and

in response to the thumb-swipe or snap gesture, taking a second action mapped to the thumb-swipe or snap gesture, wherein the second action comprises stopping recording and sending the response to the received message.

2 . The method of claim 1 , wherein the pinch gesture is in relation to the event trigger by the pinch gesture being directed at a UI display provided in response to the event trigger showing an indication of the received message.

3 . The method of claim 1 , wherein;

a third action is mapped to a pinch release gesture, wherein the third action comprises opening a message editing dialog; and

the method further comprises:

receiving, from the gesture-tracking system, the pinch release gesture.

4 . The method of claim 1 , wherein;

a third action is mapped a tug left gesture, wherein the third action comprises discarding the response unsent; and

the method further comprises:

receiving, from the gesture-tracking system, the tug left gesture.

5 . The method of claim 1 , wherein the second action of stopping recording and sending the response to the received message is mapped to the thumb-swipe or snap gesture due to a previous determination that stopping recording and sending the response is the action most commonly taken following a pinch gesture in relation to receiving a message.

6 . A computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process for an artificial reality system to respond to gestural input, the process comprising:

receiving, from a gesture-tracking system, a pinch gesture in relation to an event trigger, wherein the pinch gesture is received in response to the gesture-tracking system detecting a tip of a thumb of a user and a tip of one other finger of the user touching each other;

determining a first action, of multiple actions associated with the pinch gesture, based on a mapping of A) a context determined when the pinch gesture was made to B) the first action of the multiple actions associated with the pinch gesture, wherein the context includes one or more of a discernable state of the user, surroundings of the user, and one or more recent actions of the user relative to when the pinch gesture was made;

in response to the pinch gesture, taking the first action mapped to the pinch gesture in the context;

receiving, from the gesture-tracking system, a thumb-swipe or snap gesture, wherein the thumb-swipe or the snap gesture is received in response to the gesture-tracking system detecting either A) that the user has moved, above a threshold speed, the tip of the thumb away and to the side from the tip of the one other finger or B) that the user has moved the tip of the thumb away from the tip of the one other finger and along a side of the one other finger; and

in response to the thumb-swipe or snap gesture, taking a second action mapped to the thumb-swipe or snap gesture.

7 . The computer-readable storage medium of claim 6 , wherein:

the event trigger is an indication of a received message,

the first action mapped to the pinch gesture is recording a response to the received message, and

the second action mapped to the thumb-swipe or snap gesture is stopping recording and sending the response to the received message.

8 . The computer-readable storage medium of claim 6 , wherein the second action comprises sending a recorded message.

9 . The computer-readable storage medium of claim 6 , wherein the second action is mapped to the thumb-swipe or snap gesture due to a previous determination that the second action is the action most commonly taken, by the user, following a pinch gesture in relation to the event trigger.

10 . The computer-readable storage medium of claim 6 , wherein the context comprises one or more of:

a determination of where the user is looking;

a determination of what actions the user is taking;

a determination of who has been identified to be in the surroundings of the user; and

a history or relationship between the user and a second user related to the event trigger.

11 . The computer-readable storage medium of claim 6 , wherein:

a third action is mapped to a tug gesture, wherein the third action comprises causing a delete, discard, or cancel action for a result of the first action; and

the instructions, when executed by the computing system, further cause the computing system to perform:

receiving, from the gesture-tracking system, the tug gesture.

12 . The computer-readable storage medium of claim 6 , wherein:

a third action is mapped to a tug gesture, wherein the third action is associated with a second most common action following the first action; and

the instructions, when executed by the computing system, further cause the computing system to perform:

receiving, from the gesture-tracking system, the tug gesture.

13 . The computer-readable storage medium of claim 6 , wherein:

a third action is mapped to a pinch release gesture, wherein the third action comprises causing activation of an editing mode for a result of the first action; and

the instructions, when executed by the computing system, further cause the computing system to perform:

receiving, from the gesture-tracking system, the pinch release gesture.

14 . The computer-readable storage medium of claim 6 , wherein:

the event trigger is an indication of a calendar event alert,

the first action mapped to the pinch gesture is to bring up a map for a location in relation to the calendar event, and

the second action mapped to the thumb-swipe or snap gesture is to start navigation to the location.

15 . The computer-readable storage medium of claim 6 , wherein:

the event trigger is an indication of another person in an artificial reality environment provided by the artificial reality system;

the first action mapped to the pinch gesture is bringing up a UI related to the other person, and

the second action mapped to the thumb-swipe or snap gesture is initiating a messaging interface with that other person.

16 . An artificial reality computing system for responding to gestural input, the artificial reality computing system comprising:

one or more processors; and

one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process comprising:

receiving, from a gesture-tracking system, a pinch gesture in relation to an event trigger, wherein the pinch gesture is received in response to the gesture-tracking system detecting a tip of a thumb of a user and a tip of one other finger of the user touching each other;

determining a first action, of multiple actions associated with the pinch gesture, based on a mapping of A) a context determined when the pinch gesture was made to B) the first action of the multiple actions associated with the pinch gesture, wherein the context includes one or more of a discernable state of the user, surroundings of the user, and one or more recent actions of the user relative to when the pinch gesture was made;

in response to the pinch gesture, taking the first action mapped to the pinch gesture in the context;

receiving, from the gesture-tracking system, a thumb-swipe or snap gesture, wherein the thumb-swipe or the snap gesture is received in response to the gesture-tracking system detecting either A) that the user has moved, above a threshold speed, the tip of the thumb away and to the side from the tip of the one other finger or B) that the user has moved the tip of the thumb away from the tip of the one other finger and along a side of the one other finger; and

in response to the thumb-swipe or snap gesture, taking a second action mapped to the thumb-swipe or snap gesture.

17 . The artificial reality computing system of claim 16 , wherein:

the event trigger is an indication of a received message,

the first action mapped to the pinch gesture is recording a response to the received message, and

the second action mapped to the thumb-swipe or snap gesture is stopping recording and sending the response to the received message.

18 . The artificial reality computing system of claim 16 , wherein the second action is mapped to the thumb-swipe or snap gesture due to a previous determination that the second action is the action most commonly taken, by multiple users, following a pinch gesture in relation to the event trigger.

19 . The artificial reality computing system of claim 16 , wherein:

the event trigger is an indication of a calendar event alert,

the first action mapped to the pinch gesture is to bring up a map for a location in relation to the calendar event, and

the second action mapped to the thumb-swipe or snap gesture is to start navigation to the location.

20 . The artificial reality computing system of claim 16 , wherein:

the event trigger is an indication of another person in an artificial reality environment provided by the computing system;

the first action mapped to the pinch gesture is bringing up a UI related to the other person, and

the second action mapped to the thumb-swipe or snap gesture is initiating a messaging interface with that other person.

Assignments (1)

Continuity (1)

References Cited (400)