Systems and methods for decoding intended speech from neuronal activity
Systems and methods for decoding indented speech from neuronal activity in accordance with embodiments of the invention are illustrated. One embodiment includes a neuronal speech system for decoding intended speech from neuronal signals includes a neuronal signal recorder implanted into a user's brain, including a multielectrode array, controller circuitry, and a communication circuitry capable of transmitting data to a neuronal signal decoder, the neuronal signal decoder located externally from the user's body, including a processor, an input/output interface, and a memory, where the memory contains a neuronal speech application that directs the processor to obtain neuronal signal data from the neuronal signal recorder, where the neuronal signal data describes neuronal activity proximal to the implanted neuronal signal recorder, decode speech features from the neuronal signal data using a neuronal decoding model, construct a speech string from the decoded speech features; and output the constructed speech string via an output device.
1. A neuronal speech system for decoding intended speech from neuronal signals, comprising:
a neuronal signal recorder configured to be implanted into a human user's brain, comprising:
an array of microelectrodes, each microelectrode in the array of microelectrodes providing at least one channel for measurement of neuronal activity;
a voltage threshold filtering circuitry configured to:
apply a band-pass filter to each measurement; and
apply a voltage threshold to the band-pass filtered measurements of between −3× and −5× of root mean squared voltage of the at least one channel to produce a binary value indicating the presence or absence of an action potential (AP) for the at least one channel;
a controller circuitry, where the controller circuitry is configured to:
record local field potentials (LFPs) and APs from each microelectrode in the array of microelectrodes via the voltage threshold filtering circuitry; and
bin features describing the LFPs and APs from each microelectrode in the array of microelectrodes in a plurality of bins, where each covers a time window between 5 ms and 100 ms;
a communication circuitry configured to wirelessly transmit values of the bins as neuronal signal data to a neuronal signal decoder;
the neuronal signal decoder configured to be located external to the user's body, comprising:
a processor;
an input/output interface; and
a memory, where the memory contains a neuronal speech application that directs the processor to:
obtain neuronal signal data from the neuronal signal recorder, where the neuronal signal data describes neuronal activity proximate to the implanted neuronal signal recorder;
decode speech features from the neuronal signal data using a neuronal decoding machine learning model;
construct a speech string from the decoded speech features; and
output the constructed speech string via an output device.
2. The neuronal speech system of claim 1 , wherein to decode speech features from the neuronal signal data, the neuronal speech application further directs the processor to use a language model.
3. The neuronal speech system of claim 2 , wherein the language model adapts to the user's speech patterns.
4. The neuronal speech system of claim 1 , wherein the neuronal decoding machine learning model is a neural network.
5. The neuronal speech system of claim 1 , wherein the neuronal signal recorder is implanted below the surface of the user's brain.
6. The neuronal speech system of claim 1 , wherein the neuronal signal recorder is implanted into the hand/arm region of the motor cortex of the user's brain.
7. The neuronal speech system of claim 1 , wherein the array of microelectrodes is between 1 mm 2 and 10 mm 2 .
8. The neuronal speech system of claim 1 , wherein the speech features are phonemes.
9. The neuronal speech system of claim 1 , wherein the speech features are articulation features.
10. The neuronal speech system of claim 1 , wherein a given decoded speech feature is selected as probable based on the presence of a previously selected decoded speech feature.
11. The neuronal speech system of claim 1 , wherein the controller circuitry is configured to calculate a moving average of a local field potential amplitude or power in a particular frequency band for at least one electrode in the array of microelectrodes.
12. The neuronal speech system of claim 1 , wherein the neuronal signal data comprises action potential information and local field potential information from at least one electrode in the array of microelectrodes.
13. The neuronal speech system of claim 1 , wherein a second neuronal signal recorder is implanted into the user's brain.
14. The neuronal speech system of claim 1 , wherein the neuronal signal recorder selectively utilizes electrodes in the array of microelectrodes for recording based on at least one quality metric.
15. The neuronal speech system of claim 1 , wherein the neuronal signal recorder is utilized for a prosthetic control system.
16. The neuronal speech system of claim 1 , wherein the output device is a vocalizer.
17. The neuronal speech system of claim 1 , wherein the output device is a text-based messaging system.
18. A method for decoding intended speech from neuronal signals, comprising:
obtaining neuronal signal data from a neuronal signal recorder implanted into a user's brain, where the neuronal signal recorder comprises:
an array of microelectrodes;
a voltage threshold filtering circuitry
controller circuitry; and
a communication circuitry;
recording local field potentials (LFPs) and action potentials (APs) from each microelectrode in the microelectrode array using the voltage threshold filtering circuitry, where the voltage threshold filtering circuitry is configured to record the APs by:
applying a band-pass filter to each measurement; and
applying a voltage threshold to the band-pass filtered measurements of between −3× and −5× of root mean squared voltage of the at least one channel to produce a binary value indicating the presence or absence of an AP for the at least one channel;
binning features describing the LFPs and APs from each microelectrode in the microelectrode array in a plurality of bins, where each covers a time window between 5 ms and 100 ms, using the controller circuitry;
wirelessly transmitting the values of the bins as neuronal signal data to a neuronal signal decoder using the communication circuitry, where the neural signal decoder is located external to the user's body;
decoding speech features from the neuronal signal data using a neuronal decoding machine learning model, using the neuronal signal decoder;
constructing a speech string from the decoded speech features using the neuronal signal decoder; and
outputting the constructed speech string via an output device communicatively coupled to the neuronal signal decoder.
19. The neuronal speech system of claim 1 , wherein to apply the band-pass filter to each measurement, the voltage threshold filtering circuitry is further configured to:
calculate a common average reference of voltage measurements produced by the array of microelectrodes;
band-pass filter mean-subtracted voltage measurements from at least one channel between 0.3 Hz and 7.5 kHz; and
high-pass filter the band-pass filtered values above 250 Hz.
20. The neuronal speech system of claim 1 , wherein the neuronal decoding machine learning model is a support vector machine.