Multi-modal neural interfacing for prosthetic devices
Methods and systems to interface between physiological devices and a prosthetic device, including to receive a plurality of types of physiological activity signals from a user, decode a user movement intent from each of the plurality of signals types, and fuse the movement intents into a joint decision to control moveable elements of the prosthetic device.
1. A neural interface system comprising:
a prosthetic device;
a plurality of types of sensors configured to receive a plurality of types of physiological activity signals from a prosthetic device user;
a neural interface configured to decode a user movement intent from each of the plurality of signals types, and fuse the movement intents into a joint decision to control moveable elements of the prosthetic device, the neural interface comprising:
a plurality of classifier modules, each classifier module associated with a corresponding one of the signal types and configured to determine a user movement state from signals of the signal type;
a plurality of decoders, each associated with a corresponding one of the signal types and configured to decode a movement intent from signals of the signal type and from one or more of the user movement states; and
a fusion module to fuse movement intents from a plurality of the decoders into the joint movement decision;
wherein the plurality of signal types include a combination of two or more of,
a local field potential (LFP) signal,
a unit activity (spike) signal,
an epidural electrocorticography grid (ECoG) signal,
an electromyography (EMG) signal,
an electroencephalography (EEG) signal, and
an electronystagmography (ENG) signal.
2. The system of claim 1 , further including:
a plurality of groups of the classifier modules, each group associated with a corresponding group of control (GOC) of the prosthetic device;
a plurality of groups of the decoders, each group associated with a corresponding one of the GOC;
a plurality of fusion modules, each associated with a corresponding one of the GOCs to fuse the movement intents from decoders of the GOC into a joint decision of the GOC; and
a motion estimator to generate a movement action from joint movement decisions of a plurality of the GOCs.
3. The system of claim 2 , wherein:
the prosthetic device includes a prosthetic arm and hand; and
the groups of control include an upper arm group, a wrist group, a hand and finger group, and an endpoint group.
4. The system of claim 2 , wherein the plurality of decoders comprise a decoder to process cortical signals, a decoder to process peripheral nerve signals, a decoder to EMG signals, and a decoder to process prosthetic control signals, the system further including:
a cortical multimodal control unit (cMCU), including the decoder to process the cortical signals;
a peripheral nerve multimodal control unit (pMCU), including the decoder to process the peripheral nerve signals; and
a neural fusion unit (NFU), including the fusion module and the decoder to process the EMG and CPC signals.
5. The system of claim 1 , wherein at least one of the classifier modules is configured to determine the movement state as one of motionless, pre-movement, and peri-movement.
6. The system of claim 1 , wherein the fusion module is configured to perform one or more of decision fusion and data fusion.
7. The system of claim 1 , wherein the fusion module is integrated within the decoder.
8. The system of claim 1 , further including:
a plurality of pre-processors, each associated with a corresponding one of the signal types to receive and pre-process signals of the signal type and to selectively direct subsets of the signals to one or more of the classifier modules.
9. The system of claim 8 , wherein at least one of the pre-processors includes a multi-stage data validation module, each stage to identify signals as one of valid and invalid as a function of a corresponding metric, and wherein the validation module is configured to identify a validation stage at which a signal is identified as invalid.
10. The system of claim 1 , wherein:
the neural interface system is configured to receive and incorporate sensory feedback from the prosthetic device into the joint movement decision; and
the sensory feedback includes one or more of velocity, speed, force, direction, position, and temperature information.
11. The system of claim 1 , wherein the plurality of sensors comprises one or more relatively non-invasive physiological sensors or one or more relatively invasive physiological sensors, and
wherein the neural interface system includes a plurality of modular and configurable components, including a base configuration to process signals from the one or more relatively non-invasive physiological sensors, and one or more selectively enabled modules to process signals from the one or more relatively invasive physiological sensors.
12. The system of claim 1 , wherein:
at least one of the decoders is configured to compute an unnormalized log posterior probability (ULPP) value for each of a plurality of classes of movement in accordance with Bayesian classifiers, determine the movement intent from the ULPP values, and output the movement intent and the ULPP values; and
the fusion module is configured to generate the joint movement decision based at least in part of the ULPP values.