IP Library Granted Patent US 11,587,242
Granted Patent B1
US 11,587,242 · App. 17/408,339 · Granted Feb 21, 2023

Real-time processing of handstate representation model estimates

Inventors: Tudor Giurgica-Tiron (Stanford, CA); Adam Al-natsheh (New York, NY); Nathan Danielson (New York, NY)
Assignee: Meta Platforms Technologies, LLC
G06T7/251G06F3/011G06F3/015G06F3/017G06N20/00G06T5/002G06T13/40G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,587,242
App. No.
17/408,339
Granted
Feb 21, 2023
Kind
B1
Abstract

System and methods are provided for providing a dynamically-updated musculoskeletal representation of a hand. The system includes a plurality of neuromuscular sensors configured to continuously record a plurality of neuromuscular signals from a user, and at least one computer processor programmed to provide as input to a trained statistical model, the plurality of neuromuscular signals and temporally smooth in real-time an output of the trained statistical model. The system is also programmed to determine, based on the smoothed output of the trained statistical model, position information describing a spatial relationship between two or more connected segments of the musculoskeletal representation, force information describing a force exerted by at least one segment of the musculoskeletal representation, and update the musculoskeletal representation of the hand based, at least in part, on the position information and the force information.

Claims (46)

1. A computer-implemented method comprising:

receiving one or more neuromuscular signals from a plurality of neuromuscular sensors associated with a user; and

providing, as input to a trained statistical model, the one or more neuromuscular signals;

temporally smoothing one or more outputs of the trained statistical model;

predicting, based on the smoothed output of the trained statistical model, a probability that a musculoskeletal representation of the user's body is characterized by a specified set of position characteristics that describe how two or more connected segments of the user's body move in relation to each other within the musculoskeletal representation;

determining, based on the predicted probability, how the two or more connected segments of the user's body are to move in relation to each other; and

updating the musculoskeletal representation based, at least in part, on the determined movements.

2. The computer-implemented method of claim 1 , wherein the position characteristics that that describe how two or more connected segments of the user's body move in relation to each other within the musculoskeletal representation comprise an indication of joint type.

3. The computer-implemented method of claim 1 , wherein the position characteristics that that describe how two or more connected segments of the user's body move in relation to each other within the musculoskeletal representation comprise an indication of joint angle.

4. The computer-implemented method of claim 1 , wherein the position characteristics that that describe how two or more connected segments of the user's body move in relation to each other within the musculoskeletal representation comprise an indication of which user is providing the received neuromuscular signals.

5. The computer-implemented method of claim 1 , wherein one or more specific ways in which the user moves their body are incorporated into the trained statistical model.

6. The computer-implemented method of claim 1 , wherein the trained statistical model provides an output that indicates the likelihood that the musculoskeletal representation of the user's body is characterized by the specified set of musculoskeletal position characteristics.

7. The computer-implemented method of claim 1 , further comprising:

determining the accuracy of the trained statistical model; and

temporally smoothing, responsive to the determined accuracy of the trained statistical model, the output of the trained statistical model in real-time.

8. The computer-implemented method of claim 7 , further comprising determining at least one of an input accuracy and an output accuracy of the trained statistical model.

9. The computer-implemented method of claim 1 , wherein temporally smoothing the output of the trained statistical model comprises including in the trained statistical model at least one temporal smoothing characteristic.

10. The computer-implemented method of claim 9 , wherein the trained statistical model that includes the at least one temporal smoothing characteristic trains the statistical model with a penalization term to promote smoothness in the model outputs.

11. The computer-implemented method of claim 9 , wherein the trained statistical model that includes the at least one temporal smoothing characteristic trains the statistical model with an error function that includes an error in estimates of time derivatives of the output of the statistical model.

12. The computer-implemented method of claim 9 , wherein the trained statistical model that includes the at least one temporal smoothing characteristic includes in the trained statistical model a prior on the smoothness of the output of the statistical model.

13. A system comprising:

at least one physical processor; and

physical memory comprising computer-executable instructions that, when executed by the physical processor, cause the physical processor to:

receive one or more neuromuscular signals from a plurality of neuromuscular sensors associated with a user; and

provide, as input to a trained statistical model, the one or more neuromuscular signals;

temporally smooth one or more outputs of the trained statistical model;

predict, based on the smoothed output of the trained statistical model, a probability that a musculoskeletal representation of the user's body is characterized by a specified set of position characteristics that describe how two or more connected segments of the user's body move in relation to each other within the musculoskeletal representation;

determine, based on the predicted probability, how the two or more connected segments of the user's body are to move in relation to each other; and

update the musculoskeletal representation based, at least in part, on the determined movements.

14. The system of claim 13 , wherein temporally smoothing the outputs of the trained statistical model comprises processing the output of the trained statistical model using at least one filter.

15. The system of claim 13 , wherein temporally smoothing the output of the trained statistical model further comprises generating the output of the trained statistical model for a first timepoint based on first neuromuscular signals recorded at the first timepoint and second neuromuscular signals recorded at at least one second timepoint after the first timepoint.

16. The system of claim 13 , wherein the at least one physical processor is further programmed to:

determine a quality of the smoothed output of the trained statistical model at a first timepoint; and

adjust at least one smoothing parameter based on the determined quality; and

temporally smooth output of the trained statistical model at a second timepoint based on the at least one adjusted smoothing parameter.

17. The system of claim 13 , wherein the at least one physical processor is further programmed to temporally smooth the output of the trained statistical model based, at least in part, on at least one characteristic of a motion of the user during recording of the plurality of neuromuscular signals.

18. The system of claim 13 , further comprising:

at least one inertial measurement unit configured to continuously record a plurality of inertial measurement unit signals, wherein the at least one characteristic of a motion of the user is determined based, at least in part, on the plurality of inertial measurement unit signals.

19. The system of claim 18 , wherein the at least one characteristic of a motion of the user comprises a speed at which a user is moving a hand and/or arm of the user.

20. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:

receive one or more neuromuscular signals from a plurality of neuromuscular sensors associated with a user; and

provide, as input to a trained statistical model, the one or more neuromuscular signals;

temporally smooth one or more outputs of the trained statistical model;

predict, based on the smoothed output of the trained statistical model, a probability that a musculoskeletal representation of the user's body is characterized by a specified set of position characteristics that describe how two or more connected segments of the user's body move in relation to each other within the musculoskeletal representation;

determine, based on the predicted probability, how the two or more connected segments of the user's body are to move in relation to each other; and

update the musculoskeletal representation based, at least in part, on the determined movements.

Assignments (3)
CHANGE OF NAME Recorded May 27, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060203/0228 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 21, 2021
From: GIURGICA-TIRON, TUDOR; AL-NATSHEH, ADAM; DANIELSON, NATHAN
To: CTRL-LABS CORPORATION
Reel/Frame 057870/0257 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 21, 2021
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 057888/0769 →