IP Library Granted Patent US 10,409,371
Granted Patent B2
US 10,409,371 · App. 15/659,018 · Granted Sep 10, 2019

Methods and apparatus for inferring user intent based on neuromuscular signals

Inventors: Patrick Kaifosh (New York, NY); Timothy Machado (Palo Alto, CA); Thomas Reardon (New York, NY); Erik Schomburg (Brooklyn, NY)
Assignee: CTRL-labs Corporation
G06F3/015A61B5/0488A61B5/7267G06N7/005G06N20/00A61B5/7282
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,409,371
App. No.
15/659,018
Granted
Sep 10, 2019
Kind
B2
Abstract

Methods and system for predicting the onset of a motor action using neuromuscular signals. The system comprises a plurality of sensors configured to continuously record a plurality of neuromuscular signals from a user and at least one computer processor programmed to provide as input to a trained statistical model, the plurality of neuromuscular signals or information based on the plurality of neuromuscular signals, predict, based on an output of the trained statistical model, whether an onset of a motor action will occur within a threshold amount of time; and send a control signal to at least one device based, at least in part, on the output probability, wherein the control signal is sent to the at least one device prior to completion of the motor action by the user.

Claims (44)

1. A control system, comprising:

a plurality of sensors configured to continuously record a plurality of neuromuscular signals from a user; and

at least one computer processor programmed to:

provide as input to a trained statistical model, the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals, wherein the trained statistical model was trained based, at least in part, on neuromuscular data recorded during at least one previous performance of a motor action by one or more users and result data indicating an outcome of the motor action performed by the one or more users, wherein the trained statistical model is configured to predict the onset of the motor action prior to completion of the motor action;

predict, based on an output of the trained statistical model, whether an onset of the motor action will occur within a threshold amount of time; and

send a control signal to at least one device based, at least in part, on the prediction, wherein the control signal is sent to the at least one device prior to completion of the motor action by the user to control an operation of the at least one device.

2. The control system of claim 1 , wherein the output of the trained statistical model comprises a probability that the onset of the motor action will occur within the threshold amount of time, and

wherein predicting that an onset of the motor action will occur within a threshold amount of time comprises determining whether the probability is greater than an operating threshold.

3. The control system of claim 2 , wherein the at least one computer processor is further programmed to determine the operating threshold based, at least in part, on an analysis of data obtained from neuromuscular signals recorded by at least some of the plurality of sensors.

4. The control system of claim 3 , wherein determining the operating threshold comprises:

determining a mean anticipation time based on the data; and

determining the operating threshold based, at least in part, on the mean anticipation time.

5. The control system of claim 2 , wherein the at least one computer processor is further programmed to receive at least one performance characteristic for a task to be performed, and wherein the operating threshold is determined based, at least in part, on the received at least one performance characteristic.

6. The control system of claim 5 , wherein the at least one performance characteristic includes a performance characteristic selected from the group consisting of a false positive rate, a false negative rate, a true positive rate, a true negative rate, a positive predictive value, a negative predictive value, and a mean anticipation time.

7. The control system of claim 1 , wherein the output of the trained statistical model comprises a set of probabilities, each probability in the set indicating a probability that the onset of a respective motor action of a plurality of motor actions will occur within the threshold amount of time, and

wherein predicting that an onset of the motor action will occur within a threshold amount of time comprises determining whether any of the probabilities in the set of probabilities is greater than an operating threshold stored by the system.

8. The control system of claim 1 , wherein the output of the trained statistical model comprises a probability that the onset of a sequence of motor actions will occur within the threshold amount of time, and

wherein predicting that an onset of the motor action will occur within a threshold amount of time comprises determining whether the probability is greater than an operating threshold.

9. The control system of claim 1 , wherein the at least one computer processor is further programmed to retrain the trained statistical model based, at least in part, on at least some of the plurality of neuromuscular signals.

10. The control system of claim 1 , wherein the trained statistical model comprises a recurrent neural network.

11. The control system of claim 1 , wherein the at least one computer processor is further programmed to:

determine a set of features based on the plurality of neuromuscular signals; and

provide as input to a trained statistical model, the set of features as information based on the plurality of neuromuscular signals,

wherein predicting, based on an output of the trained statistical model, that an onset of the motor action will occur within a threshold amount of time comprises classifying the set of features using the trained statistical model.

12. The control system of claim 1 , wherein the plurality of sensors are arranged on a wearable device configured to be worn on or around a body part of the user.

13. The control system of claim 12 , wherein the wearable device comprises a flexible or elastic band configured to be worn around the body part of the user.

14. The control system of claim 13 , wherein the wearable device comprises an armband configured to be worn around an arm of the user.

15. The control system of claim 1 , further comprising hardware circuitry and/or software configured to process the neuromuscular signals prior to being provided as input to the trained statistical model, wherein processing the neuromuscular signals comprises processing selected from the group consisting of amplification, filtering, and rectification.

16. The control system of claim 1 , wherein sending the control signal to the at least one device prior to completion of the motor action by the user comprises sending the control signal at least 25 ms prior to the completion of the motor action by the user.

17. The control system of claim 1 , wherein the at least one device comprises at least one device selected from the group consisting of a video game system, a consumer electronics device, a manned vehicle, an unmanned vehicle, a robot, and a weapon.

18. The control system of claim 1 , wherein the plurality of sensors comprises sensors selected from the group consisting of electromyography (EMG) sensors, mechanomyography (MMG) sensors, and sonomyography (SMG) sensors.

19. A method of predicting an onset of a motor action, the method comprising

providing, as input to a trained statistical model, a plurality of neuromuscular signals recorded from a plurality of sensors arranged on or around a part of a user's body, wherein the trained statistical model was trained based, at least in part, on neuromuscular data recorded during at least one previous performance of a motor action by one or more users and result data indicating an outcome of the motor action performed by the one or more users, wherein the trained statistical model is configured to predict the onset of the motor action prior to completion of the motor action;

predicting, using at least one computer processor, whether an onset of the motor action will occur within a threshold amount of time; and

sending a control signal to at least one device based, at least in part, on the prediction, wherein the control signal is sent to the at least one device prior to completion of the motor action by the user to control an operation of the at least one device.

20. A computer system for training a statistical model to predict an onset of a motor action based, at least in part, on neuromuscular signal data, the computer system comprising:

an input interface configured to:

receive the neuromuscular signal data recorded during performance of a motor action performed by one or more users;

receive result data indicating an outcome of the motor action performed by the one or more users; and

at least one storage device configured to store a plurality of instructions that, when executed by at least one computer processor perform a method of:

generating training data based, at least on part, on the received neuromuscular signal data and the received result data;

training the statistical model using at least some of the generated training data to output a trained statistical model;

determining an operating threshold used to interpret output of the trained statistical model; and

storing, by the at least one storage device, the trained statistical model and the operating threshold, wherein the trained statistical model is configured to predict the onset of the motor action prior to completion of the motor action by a user.

Assignments (5)
CHANGE OF NAME Recorded May 26, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060199/0876 →
CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY DATA PREVIOUSLY RECORDED AT REEL: 051649 FRAME: 0001. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Feb 10, 2020
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 051867/0136 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 20, 2020
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, INC.
Reel/Frame 051649/0001 →
CHANGE OF NAME Recorded Oct 11, 2017
From: COGNESCENT CORPORATION
To: CTRL-LABS CORPORATION
Reel/Frame 044278/0190 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 30, 2017
From: KAIFOSH, PATRICK; MACHADO, TIMOTHY; REARDON, THOMAS; SCHOMBURG, ERIK
To: COGNESCENT CORPORATION
Reel/Frame 043445/0584 →
Continuity (2)
Provisional Application 62366419 · Jul 25, 2016
Related Publication 20180024634A1 · Jan 25, 2018
Cited By (6)
US 12,504,816 US 12,504,819 US 12,554,325 US 12,579,768 US 12,591,304 US 12,704,906