Systems and methods for using silent speech in a user interaction system
The techniques described herein relate to computerized methods and systems for integrating with a knowledge system. In some embodiments, a user interaction system may include a speech input device wearable on a user and configured to receive an electronic signal indicative of a user's speech muscle activation patterns when the user is speaking. In some embodiments, the electronic signal may include EMG data received from an EMG sensor on the speech input device. The system may include at least one processor configured to use a speech model and the electronic signal as input to the speech model to generate a text prompt. The at least one processor may use a knowledge system to take an action or generate a response based on the text prompt. In some embodiments, the system may provide context to the knowledge system.
1 . A user interaction system comprising:
a wearable speech input device configured to measure a signal indicative of a user's speech muscle activation patterns when the user is speaking; and
at least one processor configured to:
use a speech model and the signal as an input to the speech model to generate an output; and
use a knowledge system to take an action or generate a response based on the output.
2 . The system of claim 1 , wherein:
the speech input device includes an electromyography (EMG) sensor; and
the signal is an EMG signal captured from the EMG sensor when the user is silently speaking or whispering.
3 . The system of claim 2 , wherein the signal is recorded non-invasively from one or more regions of face and/or neck of the user via a conductive electrode coupled to an electronic amplifier system.
4 . The system of claim 2 , wherein silently speaking produces a sound volume less than or equal to 30 dB, wherein the sound volume is measured about 10 cm from a mouth of the user.
5 . The system of claim 1 further comprising one or more additional sensors configured to measure respective signals when the user is speaking.
6 . The system of claim 1 , wherein the output of the speech model comprises a text prompt and/or encoded features.
7 . The system of claim 6 , wherein the text prompt corresponds to one or more words spoken by the user.
8 . The system of claim 6 , wherein the text prompt includes two or more candidate transcripts of utterance of the user comprising one or more spoken words, wherein the two or more candidate transcripts are generated by the speech model.
9 . The system of claim 6 , wherein the encoded features include a probability distribution of different text tokens associated with a decoder of the speech model.
10 . The system of claim 1 further comprising an output device configured to output the response to a user, wherein outputting comprises one or more of:
outputting the response in a text format;
outputting the response in an auditory signal;
outputting the response in an image format;
outputting the response in a video format;
outputting the response in augmented reality; or
outputting the response in haptics.
11 . The system of claim 1 , wherein the at least one processor is further configured to:
determine an application based on the output of the speech model; and
cause the knowledge system to use the application to take the action or generate the response based on the output of the speech model.
12 . The system of claim 11 , further comprising a graphical user interface, wherein using the application comprises opening the application on the graphical user interface to receive user input.
13 . The system of claim 1 , wherein the speech model is trained for predicting speech label in a first domain using at least training data in a second domain different from the first domain.
14 . The system of claim 1 , wherein the speech model is trained for predicting speech label using voiced speech training data and silent speech training data.
15 . The system of claim 14 , wherein the knowledge system includes a machine learning foundation model.
16 . The system of claim 15 , wherein the machine learning foundation model includes a large language model.
17 . The system of claim 15 , wherein:
the output of the speech model comprises a text prompt; and
the machine learning foundation model is configured to use the text prompt as input and generate the response by sampling text from a distribution of possible responses.
18 . The system of claim 1 , wherein the at least one processor is configured to take the action by:
causing the knowledge system to use a machine learning foundation model to generate code based on the output of the speech model; and
executing the code to produce the action.
19 . The system of claim 1 , wherein the at least one processor is further configured to:
provide context to the knowledge system; and
use the knowledge system to take the action or generate the response additionally based on the context.
20 . The system of claim 19 , wherein the context includes personalized characteristic of the user, a location of the user, a contact list of the user, an address of the user, an email history of the user, a message history of the user, or a combination thereof.
21 . The system of claim 1 , wherein the action or the response includes prompting the user with a clarifying question.
22 . The system of claim 21 , wherein the at least one processor is further configured to:
receive a user input, the user input responsive to the clarifying question;
use the speech model and the user input to generate a text prompt; and
use the knowledge system to take additional action or generate additional response based on the text prompt.
23 . The system of claim 1 , wherein the knowledge system is configured to interact with a message system to:
generate a message; and
transmit the message over a communication network and/or output the message on an output device.
24 . The system of claim 23 , wherein the message system comprises a text messaging system, an instant messaging system, an email system, and/or a voicemail system.
25 . The system of claim 24 , wherein the at least one processor is further configured to:
provide context to the knowledge system; and
use the knowledge system to interact with the message system to generate the message additionally in combination with the context.
26 . The system of claim 25 , wherein the context comprises one or more of: a user's prior message history with an intended recipient, or a writing style of a user.
27 . The system of claim 1 , wherein the knowledge system is configured to interact with one or more applications comprising a search system, a note-taking system, a payment system, a ride-share system, a programming IDE, a web browser, or a combination thereof.
28 . A computerized method comprising:
receiving a signal indicative of a user's speech muscle activation patterns when the user is speaking;
using a speech model and the signal as an input to the speech model to generate an output; and
using a knowledge system to take an action or generate a response based on the output.
29 . The method of claim 28 , wherein receiving the signal indicative of the user's speech muscle activation patterns when the user is speaking comprises:
receiving the signal from a speech input device including an electromyography (EMG) sensor;
wherein the signal is an EMG signal captured from the EMG sensor when the user is silently speaking.
30 . A non-transitory computer-readable media comprising instructions that, when executed by one or more processors on a computing device, cause the one or more processors to:
receive a signal indicative of a user's speech muscle activation patterns when the user is speaking;
use a speech model and the signal as an input to the speech model to generate an output; and
use a knowledge system to take an action or generate a response based on the output.
31 . The non-transitory computer-readable media of claim 30 , wherein receiving the signal indicative of the user's speech muscle activation patterns when the user is speaking comprises:
receiving the signal from a speech input device including an electromyography (EMG) sensor;
wherein the signal is an EMG signal captured from the EMG sensor when the user is silently speaking.