IP Library Granted Patent US 10,937,414
Granted Patent B2
US 10,937,414 · App. 15/974,454 · Granted Mar 2, 2021

Systems and methods for text input using neuromuscular information

Inventors: Adam Berenzweig (Brooklyn, NY); Alan Huan Du (New York, NY); Jeffrey Scott Seely (New York, NY)
Assignee: Facebook Technologies, LLC
G10L15/14G06F3/015G06N7/00G10L13/08G10L15/063
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,937,414
App. No.
15/974,454
Granted
Mar 2, 2021
Kind
B2
Abstract

Systems and methods for text input based on neuromuscular information. The system includes a plurality of neuromuscular sensors, arranged on one or more wearable devices, wherein the plurality of neuromuscular sensors is configured to continuously record a plurality of neuromuscular signals from a user, at least one storage device configured to store one or more trained statistical models, and at least one computer processor programmed to obtain the plurality of neuromuscular signals from the plurality of neuromuscular sensors, provide as input to the one or more trained statistical models, the plurality of neuromuscular signals or signals derived from the plurality of neuromuscular signals, and determine based, at least in part, on an output of the one or more trained statistical models, one or more linguistic tokens.

Claims (60)

1. A computerized system for text input based on neuromuscular information, the system comprising:

a plurality of neuromuscular sensors, arranged circumferentially on one or more wearable devices configured to be worn around a user's lower arm, wherein the plurality of neuromuscular sensors is configured to record a plurality of neuromuscular signals from a user;

at least one storage device configured to store two or more trained statistical models; and at least one computer processor programmed to:

obtain, from the plurality of neuromuscular sensors, the plurality of neuromuscular signals;

provide as input to the two or more trained statistical models, the plurality of neuromuscular signals or signals derived from the plurality of neuromuscular signals; and

determine based, at least in part, on an output of the two or more trained statistical models, one or more linguistic tokens, wherein the two or more trained statistical models comprise a first trained statistical model for determining a musculo-skeletal representation of the user from the plurality of neuromuscular signals and a second trained statistical model, different from the first trained statistical model, that uses the determined musculo-skeletal representation from the first trained statistical model for determining the one or more linguistic tokens.

2. The computerized system of claim 1 , wherein the at least one computer processor is further programmed to:

generate, using a speech synthesizer, audio output corresponding to the determined one or more linguistic tokens; and

provide the audio output to the user or another user.

3. The computerized system of claim 1 , wherein the at least one computer processor is further programmed to:

obtain, from the user, an indication that the determined one or more linguistic tokens is incorrect; and

update the two or more trained statistical models based on the indication.

4. The computerized system of claim 3 , wherein the at least one computer processor is further programmed to:

obtain, from the user, an updated musculo-skeletal representation for a corrected linguistic token, wherein updating the two or more trained statistical models comprises updating the two or more trained statistical models based on the indication and the updated musculo-skeletal representation for the corrected linguistic token.

5. The computerized system of claim 4 , wherein the updated musculo-skeletal representation comprises information relating to at least one of a movement, a force, a pose, a gesture, or a muscle activation.

6. The computerized system of claim 1 , wherein identifying the one or more linguistic tokens comprises determining the musculo-skeletal representation of the user based, at least in part, on a first output of the first trained statistical model, and identifying the one or more linguistic tokens based, at least in part, on a second output of the second trained statistical model.

7. The computerized system of claim 6 , wherein the at least one computer processor is further programmed to:

generate a visual indication of the determined musculo-skeletal representation of the user.

8. The computerized system of claim 7 , wherein the at least one computer processor is further programmed to:

obtain, from the user, an indication that the determined musculo-skeletal representation is incorrect; and

update the first trained statistical model based on the indication.

9. The computerized system of claim 8 , wherein the at least one computer processor is further programmed to:

obtain, from the user, a corrected musculo-skeletal representation, wherein updating the first trained statistical model comprises updating the first trained statistical model based on the indication and the corrected musculo-skeletal representation.

10. The computerized system of claim 1 , wherein the at least one computer processor is further programmed to:

obtain, from the user, an indication that the determined one or more linguistic tokens is incorrect and an updated musculo-skeletal representation for a corrected linguistic token; and

update the second trained statistical model based on the indication and the updated musculo-skeletal representation for the corrected linguistic token.

11. The computerized system of claim 1 , wherein the at least one computer processor is further programmed to:

predict that at least one of the determined one or more linguistic tokens is incorrect;

obtain, from the user, an indication confirming the prediction that at least one of the determined one or more linguistic tokens is incorrect; and

update the two or more trained statistical models based on the indication.

12. The computerized system of claim 11 , wherein the indication from the user includes at least one of a key press, a voice command, the plurality of neuromuscular signals, the signals derived from the plurality of neuromuscular signals, a pose, a gesture, a force, or a handstate.

13. A computer implemented method for text input based on neuromuscular information, the method comprising:

obtaining, from a plurality of neuromuscular sensors, a plurality of neuromuscular signals, wherein the plurality of neuromuscular sensors is arranged circumferentially on one or more wearable devices configured to be worn around a user's lower arm, wherein the plurality of neuromuscular sensors is configured to record the plurality of neuromuscular signals from a user;

providing, using at least one computer processor, as input to two or more trained statistical models, the plurality of neuromuscular signals or signals derived from the plurality of neuromuscular signals; and

determining, using the at least one computer processor, based, at least in part, on an output of the two or more trained statistical models, one or more linguistic tokens, wherein the two or more trained statistical models comprises a first trained statistical model for determining a musculo-skeletal representation of the user from the plurality of neuromuscular signals and a second trained statistical model, different from the first trained statistical model, that uses the determined musculo-skeletal representation from the first strained statistical model for determining the one or more linguistic tokens.

14. The method of claim 13 , further comprising:

generating, using a speech synthesizer, audio output corresponding to the determined one or more linguistic tokens; and

providing the audio output to the user or another user.

15. The method of claim 13 , further comprising:

obtaining, from the user, an indication that the determined one or more linguistic tokens is incorrect; and

updating the two or more trained statistical models based on the indication.

16. The method of claim 15 , further comprising:

obtaining, from the user, an updated musculo-skeletal representation for a corrected linguistic token, wherein updating the two or more trained statistical models comprises updating the two or more trained statistical models based on the indication and the updated musculo-skeletal representation for the corrected linguistic token.

17. The method of claim 16 , wherein the updated musculo-skeletal representation comprises information relating to at least one of a movement, a force, a pose, a gesture, or a muscle activation.

18. The method of claim 13 , wherein identifying the one or more linguistic tokens comprises determining the musculo-skeletal representation of the user based, at least in part, on a first output of the first trained statistical model, and identifying the one or more linguistic tokens based, at least in part, on a second output of the second trained statistical model.

19. The method of claim 18 , further comprising:

generating a visual indication of the determined musculo-skeletal representation of the user.

20. The method of claim 19 , further comprising:

obtaining, from the user, an indication that the determined musculo-skeletal representation is incorrect; and

updating the first trained statistical model based on the indication.

21. The method of claim 20 , further comprising:

obtaining, from the user, a corrected musculo-skeletal representation, wherein updating the first trained statistical model comprises updating the first trained statistical model based on the indication and the corrected musculo-skeletal representation.

22. The method of claim 13 , further comprising:

obtaining, from the user, an indication that the determined one or more linguistic tokens is incorrect and an updated musculo-skeletal representation for a corrected linguistic token; and

updating the second trained statistical model based on the indication and the updated musculo-skeletal representation for the corrected linguistic token.

23. The method of claim 13 , further comprising:

predicting that at least one of the determined one or more linguistic tokens is incorrect;

obtaining, from the user, an indication confirming the prediction that at least one of the determined one or more linguistic tokens is incorrect; and

updating the two or more trained statistical models based on the indication.

24. The method of claim 23 , wherein the indication from the user include at least one of a key press, a voice command, the plurality of neuromuscular signals, the signals derived from the plurality of neuromuscular signals, a pose, a gesture, a force, or a handstate.

Assignments (6)
CHANGE OF NAME Recorded May 26, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060199/0876 →
CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE WAS ERRONEOUSLY FILED AS FACEBOOK TECHNOLOGIES, INC. PREVIOUSLY RECORDED ON REEL 051649 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded May 5, 2020
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 053210/0883 →
CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY DATA PREVIOUSLY RECORDED AT REEL: 051649 FRAME: 0001. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Feb 10, 2020
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 051867/0136 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 29, 2020
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, INC.
Reel/Frame 051653/0195 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 20, 2020
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, INC.
Reel/Frame 051649/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 10, 2018
From: BERENZWEIG, ADAM; DU, ALAN HUAN; SEELY, JEFFREY SCOTT
To: CTRL-LABS CORPORATION
Reel/Frame 046610/0483 →
Continuity (1)
Related Publication 20190348027A1 · Nov 14, 2019
Cited By (6)
US 12,236,954 US 12,504,816 US 12,554,325 US 12,579,768 US 12,591,304 US 12,704,906