Personalized gesture recognition for user interaction with assistant systems
In one embodiment, a method includes receiving a user request from a first user from a client system associated with a first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user, determining an intent corresponding to the user request based on the gesture-input by a personalized gesture-classification model associated with the first user, executing one or more tasks based on the determined intent and the speech-input, and sending instructions for presenting execution results of the one or more tasks to the client system responsive the user request.
1. A system comprising:
a microphone configured to receive a speech-input;
a camera configured to receive a gesture-input; and
circuitry configured to:
communicate a user input comprising at least one of the speech-input and the gesture-input to an external assistant system to cause the external assistant system to determine an output user intent from the user input;
in response to communicating the user input, receive information from the external assistant system for the output user intent; and
execute a task based at least in part on the received information for the output user intent determined from at least one of the speech-input and the gesture-input.
2. The system of claim 1 , wherein the circuitry is further configured to execute the task by displaying the received information.
3. The system of claim 1 , wherein the circuitry is further configured to execute the task by outputting audio for the received information.
4. The system of claim 1 , wherein the information is determined at least in part by performing automatic speech recognition on the speech-input.
5. The system of claim 1 , wherein the information is determined at least in part based on the user input and personal information of a user providing the user input.
6. The system of claim 1 , wherein the circuitry is further configured to communicate the user input to the external assistant system via a network.
7. The system of claim 6 , wherein the circuitry is further configured to receive the information from the external assistant system via the network.
8. The system of claim 1 , wherein the circuitry is further configured to determine a modality of the received information from the external assistant system at least in part based on a user profile associated with a user providing the user input.
9. The system of claim 1 , wherein the circuitry is further configured to determine a structure of the received information from the external assistant system at least in part based on a user profile associated with a user providing the user input.
10. The system of claim 1 , wherein the circuitry is further configured to execute the task at least in part based on a user profile associated with a user of the system.
11. The system of claim 10 , wherein the circuitry is further configured to determine the task at least in part based on a machine-learning model that is trained using the user profile.
12. The system of claim 11 , wherein the task comprises recommending an action to the user.
13. The system of claim 1 , wherein the circuitry is further configured to determine an intent of the gesture-input.
14. The system of claim 1 , wherein the user input comprises both the speech-input and the gesture-input.
15. The system of claim 14 , wherein the circuitry is further configured to determine an intent of the gesture-input at least in part based on the speech-input.
16. The system of claim 15 , wherein the circuitry is further configured to determine the intent of the gesture-input using a personalized gesture-classification model associated with a user providing the user input.
17. The system of claim 1 , wherein the microphone and the camera are disposed in augmented reality glasses or a virtual reality headset.
18. The system of claim 1 , wherein the microphone and the camera are enclosed in a computing device corresponding to the circuitry.
19. A method comprising:
receiving a speech-input from a microphone;
receiving a gesture-input from a camera;
communicating a user input comprising at least one of the speech-input and the gesture-input to an external assistant system to cause the external assistant system to determine an output user intent from the user input;
in response to communicating the user input, receiving information from the external assistant system for the output user intent; and
execute a task based at least in part on the received information for the output user intent determined from at least one of the speech-input and the gesture-input.
20. A non-transitory computer-readable medium comprising software that, when executed by a processor, is operable to:
receive a speech-input from a microphone;
receive a gesture-input from a camera;
communicate a user input comprising at least one of the speech-input and the gesture-input to an external assistant system to cause the external assistant system to determine an output user intent from the user input;
in response to communicating the user input, receive information from the external assistant system for the output user intent; and
execute a task based at least in part on the received information for the output user intent determined from at least one of the speech-input and the gesture-input.