IP Library Granted Patent US 11,163,361
Granted Patent B2
US 11,163,361 · App. 16/671,066 · Granted Nov 2, 2021

Calibration techniques for handstate representation modeling using neuromuscular signals

Inventors: Patrick Kaifosh (New York, NY); Tudor Giurgica-Tiron (Stanford, CA); Adam Berenzweig (Brooklyn, NY); Steven Kober (New York, NY); Adam Al-natsheh (New York, NY); Alexandre Barachant (Brooklyn, NY); Zhuo Wang (Forest Hills, NY)
Assignee: Facebook Technologies, LLC
G06F3/015A61B5/6824G06F3/017A61B5/1107A61B5/1112A61B5/1114A61B5/296A61B5/30A61B5/316
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,163,361
App. No.
16/671,066
Granted
Nov 2, 2021
Kind
B2
Abstract

Methods and apparatus for calibrating performance of one or more statistical models used to generate a musculoskeletal representation. The method comprises controlling presentation of instructions via a user interface to instruct the user to perform the at least one gesture and updating at least one parameter of the one or more statistical models based, at least in part on a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors during performance of the at least one gesture by the user.

Claims (76)

1. A computerized system comprising:

a memory storing information about a plurality of gestures designated for calibrating one or more statistical models used to generate a musculoskeletal representation;

a plurality of neuromuscular sensors configured to record neuromuscular signals of a user; and

at least one computer processor programmed to:

process a first plurality of neuromuscular signals using the one or more statistical models to generate the musculoskeletal representation;

identify, based on an output of the one or more statistical models, at least one gesture characteristic that the one or more statistical models are poor at estimating, wherein the plurality of gestures comprises:

at least one gesture that includes the at least one gesture characteristic; and

at least one additional gesture that does not include the at least one gesture characteristic; and

using the at least one gesture rather than the at least one additional gesture to calibrate performance of the one or more statistical models by updating a statistical model configuration based at least in part on a second plurality of neuromuscular signals recorded by the plurality of neuromuscular sensors during a performance of the at least one gesture and ground-truth data corresponding to the at least one gesture.

2. The computerized system of claim 1 , wherein updating the statistical model configuration comprises selecting a different statistical model to generate an updated musculoskeletal representation than the one or more statistical models used to generate the musculoskeletal representation.

3. The computerized system of claim 1 , further comprising a user interface configured to instruct the user to perform the at least one gesture, wherein the at least one computer processor is further programmed to control presentation of instructions via the user interface to instruct the user to perform the at least one gesture.

4. The computerized system of claim 1 , wherein the at least one computer processor is further programmed to update the statistical model configuration based, at least in part, on previously stored estimates of one or more of position information, force information, and neuromuscular information for the at least one gesture.

5. The computerized system of claim 1 , wherein the at least one computer processor is further programmed to control at least one external device to capture the ground-truth data during the performance of the at least one gesture.

6. The computerized system of claim 5 , wherein the at least one external device comprises an imaging device configured to capture at least one image, and wherein the ground-truth data is determined based, at least in part, on the at least one image.

7. The computerized system of claim 5 , wherein the at least one external device comprises an inertial measurement unit.

8. The computerized system of claim 5 , wherein the at least one external device comprises a plurality of external devices, each of which is configured to capture position information or force information of the ground-truth data.

9. The computerized system of claim 1 , wherein updating the statistical model configuration comprises training the one or more statistical models based, at least in part, on the second plurality of neuromuscular signals recorded during the performance of the at least one gesture.

10. The computerized system of claim 1 , wherein the one or more statistical models comprise a trained statistical model, and wherein updating the statistical model configuration comprises adjusting one or more parameters provided as inputs to the trained statistical model without retraining the trained statistical model.

11. The computerized system of claim 1 , wherein the at least one computer processor is further programmed to:

process at least some of the second plurality of neuromuscular signals using the updated statistical model configuration to generate an updated musculoskeletal representation; and

provide a feedback to the user during the performance of the at least one gesture, wherein the feedback relates to the extent of a match between the updated musculoskeletal representation and the ground-truth data recorded during the performance of the at least one gesture.

12. The computerized system of claim 11 , wherein providing the feedback to the user comprises providing a visual feedback.

13. The computerized system of claim 1 , wherein the at least one computer processor is further programmed to:

instruct the user to perform the at least one gesture as part of a calibration session; and

refrain from instructing the user to perform the at least one additional gesture during the calibration session.

14. The computerized system of claim 1 , wherein the at least one computer processor is further programmed to:

instruct the user to perform the at least one gesture while refraining from instructing the user to perform the at least one additional gesture;

detect, after instructing the user, the performance of the at least one gesture; and

record, in response to detecting the performance of the at least one gesture, the second plurality of neuromuscular signals.

15. The computerized system of claim 1 , wherein the at least one computer processor is further programmed to:

monitor, without notifying the user, for performances of the at least one gesture;

detect, without notifying the user and while monitoring for the performances of the at least one gesture, the performance of the at least one gesture; and

record, without notifying the user and in response to detecting the performance of the at least one gesture, the second plurality of neuromuscular signals.

16. The computerized system of claim 1 , wherein the at least one computer processor is further programmed to render a visual representation associated with the user's hand based on the musculoskeletal representation and/or an updated musculoskeletal representation.

17. The computerized system of claim 16 , wherein the visual representation is rendered during the performance of the at least one gesture.

18. The computerized system of claim 1 , wherein the at least one computer processor is further programmed to:

continuously track a performance of the one or more statistical models during usage of the one or more statistical models; and

initiate a calibration session when the performance deviates from an acceptable threshold by a particular amount.

19. The computerized system of claim 1 , wherein updating the statistical model configuration comprises updating a plurality of model parameters, and wherein the at least one computer processor is further programmed to:

store a user profile that includes user-specific calibration parameters, wherein the user-specific calibration parameters include a subset of the plurality of model parameters.

20. The computerized system of claim 19 , wherein the user-specific calibration parameters include at least one joint stiffness parameter.

21. A method comprising:

storing information about a plurality of gestures designated for calibrating one or more statistical models used to generate a musculoskeletal representation;

recording a first plurality of neuromuscular signals using a plurality of neuromuscular sensors;

processing the first plurality of neuromuscular signals using the one or more statistical models to generate the musculoskeletal representation;

identifying, based on an output of the one or more statistical models, at least one gesture characteristic that the one or more statistical models are poor at estimating, wherein the plurality of gestures comprises:

at least one gesture that includes the at least one gesture characteristic; and

at least one additional gesture that does not include the at least one gesture characteristic; and

using the at least one gesture rather than the at least one additional gesture to calibrate performance of the one or more statistical models by updating a statistical model configuration based; at least in part on a second plurality of neuromuscular signals recorded by the plurality of neuromuscular sensors during a performance of the at least one gesture and ground-truth data corresponding to the at least one gesture.

22. A computerized system comprising:

a memory storing information about a plurality of gestures designated for calibrating one or more statistical models used to generate a musculoskeletal representation;

a user interface configured to instruct a user to perform at least one of the plurality of gestures while wearing a wearable device having a plurality of neuromuscular sensors arranged thereon; and

at least one computer processor programmed to:

record a first plurality of neuromuscular signals using the plurality of neuromuscular sensors;

process the first plurality of neuromuscular signals using the one or more statistical models to generate the musculoskeletal representation;

identify, based on an output of the one or more statistical models, at least one gesture characteristic that the one or more statistical models are poor at estimating, wherein the plurality of gestures comprises:

at least one gesture that includes the at least one gesture characteristic; and

at least one additional gesture that does not include the at least one gesture characteristic;

instruct, via the user interface, the user to perform the at least one gesture while refraining from instructing the user to perform the at least one additional gesture; and

update at least one parameter of the one or more statistical models based, at least in part on a second plurality of neuromuscular signals recorded by the neuromuscular sensors during a performance of the at least one gesture.

23. The computerized system of claim 22 , wherein the at least one computer processor is further programmed to update at least one parameter of the one or more statistical models based, at least in part, on stored estimates of one or more of position information, force information, and neuromuscular information.

24. The computerized system of claim 22 , wherein the at least one computer processor is further programmed to:

control at least one external device to capture at least one measurement of position and/or force information during performance of the at least one gesture; and

update the at least one parameter of the one or more statistical models based, at least in part, on the at least one measurement of position and/or force information.

25. The computerized system of claim 22 , wherein updating the at least one parameter of the one or more statistical models comprises training the one or more statistical models based, at least in part, on the second plurality of neuromuscular signals recorded by the neuromuscular sensors during performance of the at least one gesture by the user.

26. The computerized system of claim 22 , wherein the one or more statistical models includes a trained statistical model, and wherein updating the at least one parameter of the one or more statistical models comprises adjusting one or more parameters provided as input to the trained statistical model without retraining the trained statistical model.

27. The computerized system of claim 22 , wherein the at least one computer processor is further programmed to provide feedback to the user, wherein the feedback indicates to the user whether the performance of the at least one gesture has been performed as instructed.

28. A method comprising:

storing information about a plurality of gestures designated for calibrating one or more statistical models used to generate a musculoskeletal representation;

recording a first plurality of neuromuscular signals using a plurality of neuromuscular sensors;

processing the first plurality of neuromuscular signals using the one or more statistical models to generate the musculoskeletal representation;

identifying, based on an output of the one or more statistical models, at least one gesture characteristic that the one or more statistical models are poor at estimating, wherein the plurality of gestures comprises:

at least one gesture that includes the at least one gesture characteristic; and

at least one additional gesture that does not include the at least one gesture characteristic;

instructing a user to perform the at least one gesture while refraining from instructing the user to perform the at least one additional gesture; and

updating at least one parameter of the one or more statistical models based, at least in part on a second plurality of neuromuscular signals recorded by the neuromuscular sensors during performance of the at least one gesture by the user.

Assignments (6)
CHANGE OF NAME Recorded May 26, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060199/0876 →
CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY DATA PREVIOUSLY RECORDED AT REEL: 051649 FRAME: 0001. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Feb 10, 2020
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 051867/0136 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 20, 2020
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, INC.
Reel/Frame 051649/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 18, 2019
From: KAIFOSH, PATRICK; BERENZWEIG, ADAM; KOBER, STEVEN; AL-NATSHEH, ADAM; BARACHANT, ALEXANDRE; WANG, ZHUO
To: CTRL-LABS CORPORATION
Reel/Frame 051517/0856 →
CHANGE OF NAME Recorded Dec 18, 2019
From: COGNESCENT CORPORATION
To: CTRL-LABS CORPORATION
Reel/Frame 051529/0381 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 18, 2019
From: GIURGICA-TIRON, TUDOR
To: COGNESCENT CORPORATION
Reel/Frame 051552/0669 →
Continuity (3)
Continuation 16257979 · Jan 25, 2019
Provisional Application 62621838 · Jan 25, 2018
Related Publication 20200125172A1 · Apr 23, 2020