Systems and methods for local automated speech-to-text processing
Systems and methods are described herein for enabling, on a local device, a voice control system that limits the amount of data needed to be transmitted to a remote server. A data structure is built at the local device to support a local speech-to-text model by receiving a query and transmitting, to a remote server over a communication network, a request for a speech-to-text transcription of the query. The transcription is received from the remote server and stored in the data structure at the local device in association with an audio clip of the query. Metadata describing the query is used to train the local speech-to-text model to recognize future instances of the query.
1. A method for supporting a local speech processing model for transcribing a voice input into executable data, the method comprising:
receiving the voice input via a voice-user interface of a local device;
based on determining the received voice input is in a particular format, converting the received voice input to a different format, wherein the converted voice input in the different format comprises less data than the received voice input in the particular format;
in response to determining that the local speech processing model cannot transcribe the converted voice input into executable data:
transmitting, to a remote server, a request for a transcription of the converted voice input; and
receiving, in response to the request, the transcription of the converted voice input and metadata corresponding to the converted voice input from the remote server;
storing, in a data structure at the local device, an entry that associates an audio clip of the converted voice input with the executable data corresponding to the transcription; and
training the local speech processing model to recognize subsequent instances of the voice input based on the metadata corresponding to the converted voice input, wherein the local speech processing model is a machine learning model.
2. The method of claim 1 , wherein the stored entry is used in recognition of subsequent voice inputs received at the voice-user interface of the local device.
3. The method of claim 1 , further comprising storing a plurality of entries in the data structure, each entry corresponding to a respective transcription, wherein each entry belongs to a static set of entries or a dynamic set of entries, the static set of entries corresponding to functions executable by the local device and the dynamic set of entries corresponding to content available from a content catalog.
4. The method of claim 3 , further comprising:
identifying, at the server, an update corresponding to the content catalog; and
in response to identifying the update, updating the data structure to include entries corresponding to the update;
wherein the training of the local speech processing model further comprises training the speech processing model to recognize additional words based on the update.
5. The method of claim 3 , wherein each entry of the static set of entries corresponds to at least one of playing, pausing, skipping, exiting, tuning, fast-forwarding, rewinding, recording, increasing volume, decreasing volume, powering on, or powering off.
6. The method of claim 3 , wherein each entry of the dynamic set of entries is a title, name, or identifier.
7. A method for supporting a local speech processing model for transcribing a voice input into executable data, the method comprising:
receiving the voice input via a voice-user interface of a local device;
in response to determining that the local speech processing model cannot transcribe the voice input into executable data:
transmitting, to a remote server, a request for a transcription of the voice input; and
receiving, in response to the request, the transcription of the voice input and metadata corresponding to the voice input from the remote server; and
storing, in a data structure at the local device, an entry that associates an audio clip of the voice input with the executable data corresponding to the transcription, wherein the entry comprises the audio clip mapped to a phoneme, wherein the phoneme is mapped to a set of graphemes, wherein the set of graphemes is mapped to a sequence of graphemes, and wherein the sequence of graphemes is mapped to the transcription.
8. The method of claim 1 , further comprising:
identifying, in the data structure, an action corresponding to the transcription, wherein the action is executable by the local device; and
performing the action at the local device.
9. The method of claim 1 , further comprising associating a plurality of audio clips with each entry in the data structure, wherein each audio clip corresponds to a respective voice input that is converted to a respective converted voice input, wherein each respective converted voice input corresponds to the entry, and wherein each respective voice input, prior to conversion, was received via the voice-user interface of the local device.
10. A system for supporting a local speech processing model for transcribing a voice input into executable data, the system comprising:
a local device; and
control circuitry configured to:
receive the voice input via a voice-user interface of the local device;
based on determining the received voice input is in a particular format, convert the received voice input to a different format, wherein the converted voice input in the different format comprises less data than the received voice input in the particular format;
in response to determining that the local speech processing model cannot transcribe the converted voice input into executable data:
transmit, to a remote server, a request for a transcription of the converted voice input; and
receive, in response to the request, the transcription of the converted voice input and metadata corresponding to the converted voice input from the remote server;
store, in a data structure at the local device, an entry that associates an audio clip of the converted voice input with the executable data corresponding to the transcription; and
train the local speech processing model to recognize subsequent instances of the voice input based on the metadata corresponding to the converted voice input, wherein the local speech processing model is a machine learning model.
11. The system of claim 10 , wherein the control circuitry is further configured to:
identify, at the server, an update corresponding to the content catalog; and
in response to identifying the update, update the data structure to include entries corresponding to the update; and
further train the speech processing model by training the speech processing model to recognize additional words based on the update.
12. The system of claim 10 , wherein the control circuitry is further configured to:
identify, in the data structure, an action corresponding to the transcription, wherein the action is executable by the local device; and
perform the action at the local device.
13. The system of claim 10 , wherein the stored entry is used in the recognition of the subsequent voice inputs received at the voice-user interface of the local device.
14. The system of claim 10 , wherein the control circuitry is further configured to store a plurality of entries in the data structure, each entry corresponding to a respective transcription, wherein each entry belongs to a static set of entries or a dynamic set of entries, the static set of entries corresponding to functions executable by the local device and the dynamic set of entries corresponding to content available from a content catalog.
15. The system of claim 14 , wherein each entry of the static set of entries corresponds to at least one of playing, pausing, skipping, exiting, tuning, fast-forwarding, rewinding, recording, increasing volume, decreasing volume, powering on, or powering off.
16. The system of claim 14 , wherein each entry of the dynamic set of entries is a title, name, or identifier.
17. The system of claim 10 , wherein the entry comprises the audio clip mapped to a phoneme, wherein the phoneme is mapped to a set of graphemes, wherein the set of graphemes is mapped to a sequence of graphemes, and wherein the sequence of graphemes is mapped to the transcription.
18. The system of claim 10 , wherein the control circuitry is further configured to associate a plurality of audio clips with each entry in the data structure, wherein each audio clip corresponds to a respective voice input that is converted to a respective converted voice input, wherein each respective converted voice input corresponds to the entry, and wherein each respective voice input, prior to conversion, was received via the voice-user interface of the local device.
19. The method of claim 1 , wherein the machine learning model is a neural network model.