Personalized gesture recognition for user interaction with assistant systems
In one embodiment, a method includes receiving a user request from a first user from a client system associated with a first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user, determining an intent corresponding to the user request based on the gesture-input by a personalized gesture-classification model associated with the first user, executing one or more tasks based on the determined intent and the speech-input, and sending instructions for presenting execution results of the one or more tasks to the client system responsive the user request.
1 . A method comprising, by a computer system:
receiving a first gesture-input of a user and a speech-input of the user;
determining a user intent based at least in part on the speech-input;
determining, based at least in part on the first gesture-input, a gesture identifier associated with the first gesture-input;
associating the gesture identifier with the user intent;
receiving a second gesture-input of the user;
determining if the second gesture-input corresponds to the gesture identifier; and
in response to determining that the second gesture-input corresponds to the gesture identifier, performing a task based at least in part on the user intent associated with the gesture identifier.
2 . The method according to claim 1 , wherein the gesture identifier corresponds to one of a pinch gesture, a swipe gesture, or an open palm gesture.
3 . The method according to claim 1 , wherein the computer system comprises smart glasses.
4 . The method according to claim 1 , wherein an assistant system performs the task, and the task comprises presenting a response to the user via the computer system.
5 . The method according to claim 4 , wherein the response comprises at least one of text, an image, a video, or an animation of a gesture.
6 . The method according to claim 4 , wherein the assistant system is external to the computer system and communicatively coupled to the computer system via a network.
7 . The method according to claim 4 , wherein the assistant system comprises an artificial neural network.
8 . The method according to claim 1 , wherein:
the first gesture-input comprises an arc motion by a hand of the user with at least two fingers extended,
the user intent corresponds to a next instruction, and
the gesture identifier is determined to be a swipe gesture.
9 . A non-transitory, computer-readable medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to:
receive a first gesture-input of a user and a speech-input of the user;
determine a user intent based at least in part on the speech-input;
determine, based at least in part on the first gesture-input, a gesture identifier associated with the first gesture-input;
associate the gesture identifier with the user intent;
receive a second gesture-input of the user;
determine if the second gesture-input corresponds to the gesture identifier; and
in response to determining that the second gesture-input corresponds to the gesture identifier, perform a task based at least in part on the user intent associated with the gesture identifier.
10 . The non-transitory, computer-readable medium according to claim 9 , wherein the gesture identifier corresponds to one of a pinch gesture, a swipe gesture, or an open palm gesture.
11 . The non-transitory, computer-readable medium according to claim 9 , wherein the computer system comprises smart glasses.
12 . The non-transitory, computer-readable medium according to claim 9 , wherein an assistant system performs the task, and the task comprises presenting a response to the user via the computer system.
13 . The non-transitory, computer-readable medium according to claim 12 , wherein the response comprises at least one of text, an image, a video, or an animation of a gesture.
14 . The non-transitory, computer-readable medium according to claim 12 , wherein the assistant system is external to the computer system and communicatively coupled to the computer system via a network.
15 . The non-transitory, computer-readable medium according to claim 12 , wherein the assistant system comprises an artificial neural network.
16 . The non-transitory, computer-readable medium according to claim 9 , wherein:
the first gesture-input comprises an arc motion by a hand of the user with at least two fingers extended,
the user intent corresponds to a next instruction, and
the gesture identifier is determined to be a swipe gesture.
17 . A smart glasses system, comprising:
a microphone;
a camera;
one or more processors communicatively coupled to the microphone and the camera; and
one or more non-transitory, computer-readable media storing instructions that, when executed by at least one of the one or more processors, cause the smart glasses system to:
receive from the camera a first gesture-input of a user and from the microphone a speech-input of the user;
determine a user intent based at least in part on the speech-input;
determine, based at least in part on the first gesture-input, a gesture identifier associated with the first gesture-input;
associate the gesture identifier with the user intent;
receive from the camera a second gesture-input of the user;
determine if the second gesture-input corresponds to the gesture identifier; and
in response to determining that the second gesture-input corresponds to the gesture identifier, perform a task based at least in part on the user intent associated with the gesture identifier.
18 . The smart glasses system according to claim 17 , wherein an assistant system performs the task, and the task comprises presenting a response to the user via the smart glasses system.
19 . The smart glasses system according to claim 18 ,
wherein the response comprises at least one of text, an image, a video, or an animation of a gesture, and
wherein the gesture identifier corresponds to one of a pinch gesture, a swipe gesture, or an open palm gesture.
20 . The smart glasses system according to claim 17 , wherein:
the first gesture-input comprises an arc motion by a hand of the user with at least two fingers extended,
the user intent corresponds to a next instruction, and
the gesture identifier is determined to be a swipe gesture.