Personalized gesture recognition for user interaction with assistant systems

1. A method comprising, by one or more computing systems:

receiving, from a client system associated with a first user, a user request from the first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user;

determining, by a personalized gesture-classification model associated with the first user, an intent corresponding to the user request based on the gesture-input;

sending, to the client system, instructions for prompting the first user for one or more user inputs;

determining, by an updated personalized gesture-classification model associated with the first user, an updated intent, wherein the updated personalized gesture-classification model is generated based on the one or more user inputs;

executing one or more tasks based on the updated intent and the speech-input; and

sending, to the client system, instructions for presenting execution results of the one or more tasks responsive the user request.

2. The method of claim 1 , further comprising training the personalized gesture-classification model for the first user, wherein the training comprises:

accessing, from a data store, a plurality of input tuples associated with the first user, wherein each input tuple comprises a gesture-input and a corresponding speech-input;

determining, by a natural-language understanding (NLU) module, a plurality of intents corresponding to the plurality of speech-inputs, respectively;

generating, for the plurality of gesture-inputs, a plurality of feature representations based on one or more machine-learning models;

determining a plurality of gesture identifiers for the plurality of gesture-inputs, respectively, based on their respective feature representations;

associating the plurality of intents with the plurality of gesture identifiers, respectively; and

generating the personalized gesture-classification model based on the plurality of feature representations of their respective gesture-inputs and the associations between the plurality of intents and their respective gesture identifiers.

3. The method of claim 2 , further comprising:

accessing, from the data store, a general gesture-classification model corresponding to a general user population, wherein training the personalized gesture-classification model is further based on the general gesture-classification model.

4. The method of claim 3 , wherein the general gesture-classification model is trained based on a plurality of gesture-inputs from the general user population.

5. The method of claim 2 , wherein the one or more machine-learning models are based on one or more of a neural network model or a long-short term memory (LSTM) model.

6. The method of claim 2 , wherein training the personalized gesture-classification model is further based on user feedback data from the first user.

7. The method of claim 1 , further comprising:

generating, by one or more automatic speech recognition (ASR) modules, one or more text-inputs for the speech-input.

8. The method of claim 7 , wherein determining the intent corresponding to the user request is further based on the one or more text-inputs of the speech-input.

9. The method of claim 1 , wherein the personalized gesture-classification model is based on convolutional neural networks.

10. The method of claim 1 , further comprising generating a feature representation for the gesture-input by the personalized gesture-classification model, wherein generating the feature representation comprises:

dividing the gesture-input into one or more components; and

modeling the one or more components into the feature representation for the gesture-input.

11. The method of claim 1 , further comprising generating a feature representation for the gesture-input by the personalized gesture-classification model, wherein generating the feature representation comprises:

determining temporal information associated with the gesture-input; and

modeling the temporal information into the feature representation for the gesture-input.

12. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

receive, from a client system associated with a first user, a user request from the first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user;

send, to the client system, instructions for prompting the first user for one or more user inputs;

determine, by an updated personalized gesture-classification model associated with the first user, an updated intent, wherein the updated personalized gesture-classification model is generated based on the one or more user inputs;

execute one or more tasks based on the updated intent and the speech-input; and

send, to the client system, instructions for presenting execution results of the one or more tasks responsive the user request.

13. The media of claim 12 , wherein the software is further operable when executed to train the personalized gesture-classification model for the first user, wherein the software is further operable when executed to:

access, from a data store, a plurality of input tuples associated with the first user, wherein each input tuple comprises a gesture-input and a corresponding speech-input;

determine, by a natural-language understanding (NLU) module, a plurality of intents corresponding to the plurality of speech-inputs, respectively;

generate, for the plurality of gesture-inputs, a plurality of feature representations based on one or more machine-learning models;

determine a plurality of gesture identifiers for the plurality of gesture-inputs, respectively, based on their respective feature representations;

associate the plurality of intents with the plurality of gesture identifiers, respectively; and

generate the personalized gesture-classification model based on the plurality of feature representations of their respective gesture-inputs and the associations between the plurality of intents and their respective gesture identifiers.

14. The media of claim 13 , wherein the software is further operable when executed to:

access, from the data store, a general gesture-classification model corresponding to a general user population, wherein training the personalized gesture-classification model is further based on the general gesture-classification model.

15. The media of claim 14 , wherein the general gesture-classification model is trained based on a plurality of gesture-inputs from the general user population.

16. The media of claim 13 , wherein the one or more machine-learning models are based on one or more of a neural network model or a long-short term memory (LSTM) model.

17. The media of claim 13 , wherein training the personalized gesture-classification model is further based on user feedback data from the first user.

18. The media of claim 12 , wherein the software is further operable when executed to:

generate, by one or more automatic speech recognition (ASR) modules, one or more text-inputs for the speech-input.

19. The media of claim 12 , wherein the personalized gesture-classification model is based on convolutional neural networks.

20. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:

receive, from a client system associated with a first user, a user request from the first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user;

send, to the client system, instructions for prompting the first user for one or more user inputs;

determine, by an updated personalized gesture-classification model associated with the first user, an updated intent, wherein the updated personalized gesture-classification model is generated based on the one or more user inputs;

execute one or more tasks based on the updated intent and the speech-input; and

send, to the client system, instructions for presenting execution results of the one or more tasks responsive the user request.