Personalized gesture recognition for user interaction with assistant systems
In one embodiment, a method includes accessing a plurality of input tuples associated with a first user from a data store, wherein each input tuple comprises a gesture-input and a corresponding speech-input, determining a plurality of intents corresponding to the plurality of speech-inputs, respectively, by a natural-language understanding (NLU) module, generating a plurality of feature representations for the plurality of gesture-inputs based on one or more machine-learning models, determining a plurality of gesture identifiers for the plurality of gesture-inputs, respectively, based on their respective feature representations, associating the plurality of intents with the plurality of gesture identifiers, respectively, and training a personalized gesture-classification model for the first user based on the plurality of feature representations of their respective gesture-inputs and the associations between the plurality of intents and their respective gesture identifiers.
1. A method comprising, by one or more computing systems:
accessing, from a data store, a plurality of input tuples associated with a first user, wherein each input tuple comprises a gesture-input and a corresponding speech-input;
determining, by a natural-language understanding (NLU) module, a plurality of intents corresponding to the plurality of speech-inputs, respectively;
generating, for the plurality of gesture-inputs, a plurality of feature representations based on one or more machine-learning models;
determining a plurality of gesture identifiers for the plurality of gesture-inputs, respectively, based on their respective feature representations;
associating the plurality of intents with the plurality of gesture identifiers, respectively; and
training, for the first user, a personalized gesture-classification model based on the plurality of feature representations of their respective gesture-inputs and the associations between the plurality of intents and their respective gesture identifiers.
2. The method of claim 1 , further comprising:
accessing, from the data store, a general gesture-classification model corresponding to a general user population, wherein training the personalized gesture-classification model is further based on the general gesture-classification model.
3. The method of claim 2 , wherein the general gesture-classification model is trained based on a plurality of gesture-inputs from the general user population.
4. The method of claim 1 , further comprising:
generating, by one or more automatic speech recognition (ASR) modules, a plurality of text-inputs for the plurality of speech-inputs, respectively.
5. The method of claim 4 , wherein determining the plurality of intents corresponding to the plurality of speech-inputs, respectively, is based on the plurality of text-inputs of the respective speech-inputs.
6. The method of claim 1 , wherein the one or more machine-learning models are based on one or more of a neural network model or a long-short term memory (LSTM) model.
7. The method of claim 1 , wherein the personalized gesture-classification model is based on convolutional neural networks.
8. The method of claim 1 , wherein generating each feature representation for each gesture-input comprises:
dividing the gesture-input into one or more components; and
modeling the one or more components into the feature representation for the gesture-input.
9. The method of claim 1 , wherein generating each feature representation for each gesture-input comprises:
determining temporal information associated with the gesture-input; and
modeling the temporal information into the feature representation for the gesture-input.
10. The method of claim 1 , further comprising:
receiving, from a client system associated with the first user, a new gesture-input from the first user; and
determining, for the new gesture-input, an intent corresponding to the new gesture-input based on the personalized gesture-classification model.
11. The method of claim 10 , further comprising:
executing one or more tasks based on the determined intent.
12. The method of claim 1 , wherein training the personalized gesture-classification model is further based on user feedback data from the first user.
13. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
access, from a data store, a plurality of input tuples associated with a first user, wherein each input tuple comprises a gesture-input and a corresponding speech-input;
determine, by a natural-language understanding (NLU) module, a plurality of intents corresponding to the plurality of speech-inputs, respectively;
generate, for the plurality of gesture-inputs, a plurality of feature representations based on one or more machine-learning models;
determine a plurality of gesture identifiers for the plurality of gesture-inputs, respectively, based on their respective feature representations;
associate the plurality of intents with the plurality of gesture identifiers, respectively; and
train, for the first user, a personalized gesture-classification model based on the plurality of feature representations of their respective gesture-inputs and the associations between the plurality of intents and their respective gesture identifiers.
14. The media of claim 13 , wherein the software is further operable when executed to:
access, from the data store, a general gesture-classification model corresponding to a general user population, wherein training the personalized gesture-classification model is further based on the general gesture-classification model.
15. The media of claim 14 , wherein the general gesture-classification model is trained based on a plurality of gesture-inputs from the general user population.
16. The media of claim 13 , wherein the software is further operable when executed to:
generate, by one or more automatic speech recognition (ASR) modules, a plurality of text-inputs for the plurality of speech-inputs, respectively.
17. The media of claim 16 , wherein determining the plurality of intents corresponding to the plurality of speech-inputs, respectively, is based on the plurality of text-inputs of the respective speech-inputs.
18. The media of claim 13 , wherein the one or more machine-learning models are based on one or more of a neural network model or a long-short term memory (LSTM) model.
19. The media of claim 13 , wherein the personalized gesture-classification model is based on convolutional neural networks.
20. A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
access, from a data store, a plurality of input tuples associated with a first user, wherein each input tuple comprises a gesture-input and a corresponding speech-input;
determine, by a natural-language understanding (NLU) module, a plurality of intents corresponding to the plurality of speech-inputs, respectively;
generate, for the plurality of gesture-inputs, a plurality of feature representations based on one or more machine-learning models;
determine a plurality of gesture identifiers for the plurality of gesture-inputs, respectively, based on their respective feature representations;
associate the plurality of intents with the plurality of gesture identifiers, respectively; and
train, for the first user, a personalized gesture-classification model based on the plurality of feature representations of their respective gesture-inputs and the associations between the plurality of intents and their respective gesture identifiers.