Systems and methods for improved speech recognition using neuromuscular information
Systems and methods for using neuromuscular information to improve speech recognition. The system includes a plurality of neuromuscular sensors arranged on one or more wearable devices and configured to continuously record a plurality of neuromuscular signals from a user, at least one storage device configured to store one or more trained statistical models for determining text based on audio input and the plurality of neuromuscular signals, at least one input interface configured to receive the audio input, and at least one computer processor programmed to obtain the audio input and the plurality of neuromuscular signals, provide as input to the one or more trained statistical models, the audio input and the plurality of neuromuscular signals or signals derived from the plurality of neuromuscular signals, and determine based, at least in part, on an output of the one or more trained statistical models, the text.
1. A computerized system for using neuromuscular information to improve speech recognition, the system comprising:
at least one input interface configured to capture speech data produced by a user;
one or more neuromuscular sensors configured to be worn on the user's wrist and detect one or more neuromuscular signals from the user's wrist corresponding to one or more movements of an arm, a wrist, and/or a hand of the user;
a memory storing a statistical model trained to:
take, as input, the one or more neuromuscular signals obtained from the user's wrist and/or information based on the one or more neuromuscular signals obtained from the user's wrist; and
output information indicative of activation states of muscles within the arm, the wrist, and/or the hand of the user; and
at least one computer processor programmed to:
obtain the speech data using the input interface;
obtain the one or more neuromuscular signals from the user's wrist;
use the statistical model to infer from the one or more neuromuscular signals obtained from the user's wrist and/or the information based on the one or more neuromuscular signals obtained from the user's wrist, an activation state of a muscle of the user's arm, wrist, or hand; and
use the inferred activation state of the muscle of the user's arm, wrist, or hand to improve recognition of the speech data.
2. The computerized system of claim 1 , wherein both the speech data and the one or more neuromuscular signals obtained from the user's wrist are simultaneously captured from the user.
3. The computerized system of claim 1 , wherein the activation state of the muscle of the user's arm, wrist, or hand comprises an activation state of one or more muscle subunits within the muscle of the user's arm, wrist, or hand.
4. The computerized system of claim 1 , wherein:
the speech data are captured before the one or more neuromuscular signals are obtained from the user's wrist;
the at least one computer processor is further programmed to:
predict a textual representation of the speech data based on the speech data; and
display, before obtaining the one or more neuromuscular signals from the user's wrist, the textual representation of the speech data to the user via a user interface; and
using the inferred activation state of the muscle of the user's arm, wrist, or hand to improve recognition of the speech data comprises:
detecting, based on the inferred activation state of the muscle of the user's arm, wrist, or hand, the user's indication of an error in speaking; and
modifying the textual representation of the speech data within the user interface in response to the user's indication.
5. The computerized system of claim 1 , wherein:
the speech data are captured before the one or more neuromuscular signals are obtained from the user's wrist;
the at least one computer processor is further programmed to:
predict a textual representation of the speech data based on the speech data; and
display, before obtaining the one or more neuromuscular signals from the user's wrist, the textual representation of the speech data to the user via a user interface; and
using the inferred activation state of the muscle of the user's arm, wrist, or hand to improve recognition of the speech data comprises:
detecting, based on the inferred activation state of the muscle of the user's arm, wrist, or hand, the user's indication of an error in predicting the textual representation of the speech data from the speech data;
predicting an alternative textual representation of the speech data; and
displaying the alternative textual representation of the speech data to the user via the user interface.
6. The computerized system of claim 1 , wherein using the inferred activation state of the muscle of the user's arm, wrist, or hand to improve recognition of the speech data comprises:
deriving contextual information from the inferred activation state of the muscle of the user's arm, wrist, or hand; and
predicting a textual representation of the speech data based on the speech data and the contextual information derived from the inferred activation state of the muscle of the user's arm, wrist, or hand.
7. The computerized system of claim 6 , wherein predicting the textual representation of the speech data comprises predicting at least a first portion of the textual representation of the speech data based on the speech data but not the contextual information.
8. The computerized system of claim 7 , wherein predicting the textual representation of the speech data comprises predicting at least a second portion of the textual representation of the speech data based on the contextual information but not the speech data.
9. The computerized system of claim 1 , wherein using the statistical model to infer the activation state of the muscle of the user's arm, wrist, or hand comprises using the one or more neuromuscular signals obtained from the user's wrist and/or the information based on the one or more neuromuscular signals obtained from the user's wrist to infer an activated state of a muscle fiber within the muscle of the user's arm, wrist, or hand and a substantially simultaneous deactivated state of an additional muscle fiber within the same muscle of the user's arm, wrist, or hand.
10. The computerized system of claim 1 , wherein the muscle is located within the user's arm.
11. A computer-implemented method for using neuromuscular information to improve speech recognition, the method comprising:
obtaining, via an input interface, speech data produced by a user;
obtaining, via one or more neuromuscular sensors worn on the user's wrist, one or more neuromuscular signals from the user's wrist corresponding to one or more movements of an arm, a wrist, and/or a hand of the user;
using a statistical model to infer, from the one or more neuromuscular signals obtained from the user's wrist and/or information based on the one or more neuromuscular signals obtained from the user's wrist, an activation state of a muscle of the user's arm, wrist, or hand, wherein the statistical model is trained to:
take, as input, the one or more neuromuscular signals obtained from the user's wrist and/or information based on the one or more neuromuscular signals obtained from the user's wrist; and
output information indicative of activation states of muscles within the arm, the wrist, and/or the hand of the user; and
using the inferred activation state of the muscle of the user's arm, wrist, or hand to improve recognition of the speech data.
12. The method of claim 11 , wherein the muscle is located within the user's arm.
13. The method of claim 11 , further comprising training the statistical model to
output information indicative of activation states of muscle subunits within the muscle of the user's arm, wrist, or hand, wherein using the statistical model to infer the activation state of the muscle of the user's arm, wrist, or hand comprises:
inputting, to the statistical model, the one or more neuromuscular signals obtained from the user's wrist and/or the information based on the one or more neuromuscular signals obtained from the user's wrist; and
inferring, from a corresponding output of the statistical model, an activation state of one or more muscle subunits within the muscle of the user's arm, wrist, or hand.
14. The method of claim 13 , wherein:
the speech data are captured before the one or more neuromuscular signals are obtained from the user's wrist;
the computer-implemented method further comprises:
predicting a textual representation of the speech data based on the speech data; and
displaying the textual representation of the speech data to the user via a user interface before obtaining the one or more neuromuscular signals from the user's wrist; and
using the inferred activation state of the muscle of the user's arm, wrist, or hand to improve recognition of the speech data comprises:
detecting, based on the inferred activation state of the muscle of the user's arm, wrist, or hand, the user's indication of an error in speaking; and
modifying the textual representation of the speech data within the user interface in response to the user's indication.
15. The method of claim 13 , wherein:
the speech data are captured before the one or more neuromuscular signals are obtained from the user's wrist;
the computer-implemented method further comprises:
predicting a textual representation of the speech data based on the speech data; and
displaying the textual representation of the speech data to the user via a user interface before obtaining the one or more neuromuscular signals from the user's wrist; and
using the inferred activation state of the muscle of the user's arm, wrist, or hand to improve recognition of the speech data comprises:
detecting, based on the inferred activation state of the muscle of the user's arm, wrist, or hand, the user's indication of an error in predicting the textual representation of the speech data from the speech data;
predicting an alternative textual representation of the speech data; and
displaying the alternative textual representation of the speech data to the user via the user interface.
16. The method of claim 11 , wherein using the inferred activation state of the muscle of the user's arm, wrist, or hand to improve recognition of the speech data comprises:
deriving contextual information from the inferred activation state of the muscle of the user's arm, wrist, or hand; and
predicting a textual representation of the speech data based on the speech data and the contextual information derived from the inferred activation state of the muscle of the user's arm, wrist, or hand.
17. The method of claim 16 , wherein predicting the textual representation of the speech data comprises determining at least a first portion of the textual representation of the speech data based on the speech data but not the contextual information.
18. The method of claim 17 , wherein predicting the textual representation of the speech data further comprises predicting at least a second portion of the textual representation of the speech data based on the contextual information derived from the one or more contemporaneous neuromuscular signals obtained from the user's wrist but not the speech data.
19. The method of claim 18 , wherein the first portion and the second portion are overlapping.
20. The method of claim 18 , wherein the first portion and the second portion are non-overlapping.
21. The method of claim 16 , wherein:
the contextual information is associated with a context in which the speech data was obtained from the user; and
predicting the textual representation of the speech data comprises determining a textual representation of the speech data most likely conveyed by the speech data in the context.
22. The method of claim 16 , wherein:
the contextual information comprises information about the one or more movements of the arm, the wrist, and/or the hand of the user; and
predicting the textual representation of the speech data comprises interpreting the speech data based on the information about the one or more movements of the arm, the wrist, and/or the hand of the user.
23. A computerized system for using neuromuscular information to improve speech recognition, the system comprising:
at least one input interface configured to capture speech data produced by a user;
one or more neuromuscular sensors configured to be worn on the user's wrist and detect one or more neuromuscular signals from the user's wrist corresponding to one or more movements of an arm, a wrist, and/or a hand of the user;
a memory storing a statistical model trained to:
take, as input, the one or more neuromuscular signals obtained from the user's wrist and/or information based on the one or more neuromuscular signals obtained from the user's wrist; and
output information indicative of activation states of muscle subunits within a muscle of the arm, the wrist, and/or the hand of the user; and
at least one computer processor programmed to:
obtain the speech data from the input interface;
obtain the one or more neuromuscular signals from the user's wrist;
use the statistical model to infer, from the one or more neuromuscular signals obtained from the user's wrist and/or the information based on the one or more neuromuscular signals obtained from the user's wrist, an activation state of one or more muscle subunits within the muscle of the arm, the wrist, and/or the hand of the user; and
use the inferred activation state of the one or more muscle subunits within the muscle of the arm, the wrist, and/or the hand of the user to improve recognition of the speech data.
24. The computerized system of claim 23 , further comprising:
a user interface configured to display visual feedback to the user, wherein the visual feedback is generated based on the inferred activation state of the one or more muscle subunits within the muscle of the arm, the wrist, and/or the hand of the user.