Granted Patent B2

US 12,625,557 · App. 18/661,520 · Granted May 12, 2026

Inventors: Shervin Ghasemlou (Redmond, WA); Devashish Prasad Joshi (Menlo Park, CA); Rongzhou Shen (Kirkland, WA); Riza Kazemi (Vancouver, CA)

Assignee: Meta Platforms, Inc.

G06F3/017G06F3/012G06F3/0346

Abstract

Claims (76)

1 . A method, comprising:

detecting, by a head-mounted device, a placement of the head-mounted device on a head of a user;

in response to detecting a user activity performed by the user and without receiving another input from the user, determining a suggestion based on the user activity;

presenting, by the head-mounted device, the suggestion;

determining, by a head-gesture detection model associated with the head-mounted device and based on at least one signal from at least one inertial measurement unit (IMU) sensor of the head-mounted device, a head gesture performed by the user;

while the user activity is ongoing and in accordance with a determination that the head-gesture performed by the user corresponds to an acceptance of the suggestion, executing, by an assistant system associated with the head-mounted device, an action to be performed by the assistant system based on the head gesture and the suggestion;

in response to detecting a conclusion of the user activity performed by the user and without receiving another input from the user, determining a different suggestion based on the conclusion of the user activity;

presenting, by the head-mounted device, the different suggestion;

determining, by the head-gesture detection model associated with the head-mounted device and based on at least one signal from at least one IMU sensor of the head-mounted device, the head gesture performed by the user; and

in accordance with a determination that the head-gesture performed by the user corresponds to an acceptance of the different suggestion, executing, by the assistant system associated with the head-mounted device, a different action to be performed by the assistant system based on the head gesture and the different suggestion.

2 . The method of claim 1 , wherein detecting the placement of the head-mounted device on the head of the user is based on a machine-learning model comprising one or more finite state machines.

3 . The method of claim 1 , wherein presenting the suggestion comprises presenting the suggestion to the user during a dialog session between the user and an assistant chatbot associated with the assistant system.

4 . The method of claim 1 , wherein

determining the suggestion is further based on contextual information associated with the user.

5 . The method of claim 1 , further comprising:

in response to detecting a second user activity, distinct from the user activity, the user and without receiving another input from the user, determining a second suggestion based on the second user activity;

presenting, by the head-mounted device, the second suggestion;

determining, by the head-gesture detection model and based on at least one second signal from the at least IMU sensor, a second head gesture performed by the user; and

while the second user activity is ongoing and in accordance with a determination that the second head-gesture performed by the user corresponds to an acceptance of the second suggestion, executing, by the assistant system, a second action to be performed by the assistant system based on the second head gesture and the second suggestion.

6 . The method of claim 1 , wherein the at least one signal from the at least one IMU sensor is sampled at a frequency not greater than 26 Hz.

7 . The method of claim 1 , wherein the head-gesture detection model is based on one or more neural networks that are operable to learn contexts and track relationships in sequential data associated with the at least one signal from the at least one IMU sensor.

8 . The method of claim 1 , wherein a size of the head-gesture detection model is no greater than 1 MB.

9 . The method of claim 1 , wherein the action comprises at least one of:

executing a task associated with the suggestion;

not executing the task associated with the suggestion;

executing the task associated with the suggestion in a particular way; and

activating the assistant system.

10 . The method of claim 1 , wherein:

the head gesture comprises an up-down sagittal head nod; and

the action comprises executing a task associated with the suggestion.

11 . The method of claim 1 , wherein:

the head gesture comprises a left-right transverse head shake; and

the action comprises not executing a task associated with the suggestion.

12 . The method of claim 1 , wherein:

the action comprises a first action of a plurality of actions to be performed by the assistant system; and

the action is selected from the plurality of actions based on the head gesture.

13 . The method of claim 12 , wherein the plurality of actions comprise at least one of:

executing a task associated with the suggestion;

not executing the task associated with the suggestion;

executing the task associated with the suggestion in a particular way; and

activating the assistant system.

14 . The method of claim 1 , further comprising:

accessing, from a database associated with the assistant system, a plurality of pairs between head gestures and indications corresponding to the head gestures; and

identifying a first pair from the plurality of pairs, wherein the first pair is between the head gesture and an indication corresponding to the head gesture, and wherein the action is selected from a plurality of actions based on the indication corresponding to the head gesture.

15 . The method of claim 14 , wherein the plurality of pairs between the head gestures and the indications corresponding to the head gestures are personalized to the user.

16 . The method of claim 14 , wherein the plurality of pairs between the head gestures and the indications corresponding to the head gestures are customized for a region with which the user is associated.

17 . The method of claim 1 , wherein the head gesture comprises at least one of:

an up-down sagittal head nod;

a left-right transverse head shake;

a one-way head tilt along a frontal plane associated with the head of the user;

a side-to-side head tilt across the frontal plane; and

a one-way transverse head rotation.

18 . The method of claim 1 , further comprising:

activating the head-gesture detection model in response to the user initiating a dialog session between the user and an assistant chatbot associated with the assistant system.

19 . A system, comprising:

one or more processors; and

a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:

detect, by a head-mounted device, a placement of the head-mounted device on a head of a user;

in response to detecting a user activity performed by the user and without receiving another input from the user, determine a suggestion based on the user activity;

present, by the head-mounted device, the suggestion;

determine, by a head-gesture detection model associated with the head-mounted device and based on at least one signal from at least one inertial measurement unit (IMU) sensor of the head-mounted device, a head gesture performed by the user;

while the user activity is ongoing and in accordance with a determination that the head-gesture performed by the user corresponds to an acceptance of the suggestion, execute, by an assistant system associated with the head-mounted device, an action to be performed by the assistant system based on the head gesture and the suggestion;

in response to detecting a conclusion of the user activity performed by the user and without receiving another input from the user, determine a second suggestion, distinct from the suggestion, based on the conclusion of the user activity;

present, by the head-mounted device, the second suggestion;

determine, by the head-gesture detection model associated with the head-mounted device and based on at least one signal from at least one IMU sensor of the head-mounted device, the head gesture performed by the user; and

in accordance with a determination that the head-gesture performed by the user corresponds to an acceptance of the second suggestion, execute, by the assistant system associated with the head-mounted device, a second action to be performed by the assistant system based on the head gesture and the second suggestion.

20 . One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

detect, by a head-mounted device, a placement of the head-mounted device on a head of a user;

in response to detecting a user activity performed by the user and without receiving another input from the user, determine a suggestion based on the user activity;

cause the head-mounted device to present the suggestion;

determine, by a head-gesture detection model associated with the head-mounted device and based on at least one signal from at least one inertial measurement unit (IMU) sensor of the head-mounted device, a head gesture performed by the user;

while the user activity is ongoing and in accordance with a determination that the head-gesture performed by the user corresponds to an acceptance of the suggestion, execute, by an assistant system associated with the head-mounted device, an action to be performed by the assistant system based on the head gesture and the suggestion;

in response to detecting a conclusion of the user activity performed by the user and without receiving another input from the user, determine a second suggestion, distinct from the suggestion, based on the conclusion of the user activity;

cause the head-mounted device to detect the second suggestion;

determine, by the head-gesture detection model associated with the head-mounted device and based on at least one signal from at least one IMU sensor of the head-mounted device, the head gesture performed by the user; and

in accordance with a determination that the head-gesture performed by the user corresponds to an acceptance of the second suggestion, execute, by the assistant system associated with the head-mounted device, a second action to be performed by the assistant system based on the head gesture and the second suggestion.

Assignments (1)

Continuity (2)

References Cited (263)