IP Library Granted Patent US 11,361,522
Granted Patent B2
US 11,361,522 · App. 16/657,029 · Granted Jun 14, 2022

User-controlled tuning of handstate representation model parameters

Inventors: Patrick Kaifosh (New York, NY); Adam Berenzweig (Brooklyn, NY); Adam Al-natsheh (New York, NY)
Assignee: Facebook Technologies, LLC
G06T19/20G06F1/163G06F3/011G06F3/014G06F3/015G06F3/017G06N7/005G06N20/10G06T15/005G06T2219/2021
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,361,522
App. No.
16/657,029
Granted
Jun 14, 2022
Kind
B2
Abstract

Methods and apparatus for enabling a user or third-party to select or adjust parameters of one or more statistical models used to generate a musculoskeletal representation. The method comprises providing as input to the statistical model(s), a plurality of neuromuscular signals recorded by a plurality of neuromuscular sensors during performance of at least one gesture by a user, wherein the at least one gesture is performed by the user while wearing a wearable device having the plurality of neuromuscular sensors arranged thereon, rendering at least one visual representation based on an output of the statistical model(s), and receiving user or third-party input to adjust parameters of the statistical model(s) based on the rendered at least one visual representation, the user input including a selection of a particular statistical model of the statistical model(s) and/or an adjustment of parameters associated with the particular statistical model.

Claims (79)

1. A method comprising:

at an electronic device that is communicatively coupled to one or more wearable neuromuscular sensors and a user interface:

receiving, from the one or more wearable neuromuscular sensors, a plurality of neuromuscular signals detected during performance of a gesture by a user;

providing, as input to one or more statistical models, information associated with the plurality of neuromuscular signals;

receiving, as one or more outputs from the one or more statistical models, pose and/or force estimates associated with a musculoskeletal representation of the user;

providing, to the user interface, a visual representation of the musculoskeletal representation using, in part, the pose and/or force estimates;

receiving, from the user interface, user input to adjust the visual representation to conform to the gesture;

based on the user input, determining a tuned statistical model for the user that predicts the gesture more accurately than the one or more statistical models when using the information associated with the plurality of neuromuscular signals as input by selecting the tuned statistical model from the one or more statistical models; and

switching one or more times between displaying the visual representation and a second candidate visual representation prior to receiving the user input.

2. The method of claim 1 , wherein determining the tuned statistical model comprises determining the statistical model that resulted in the adjusted visual representation.

3. The method of claim 1 , further comprising:

receiving, from the one or more wearable neuromuscular sensors, a plurality of additional neuromuscular signals detected during performance of an additional gesture by the user;

providing, as input to the tuned statistical model, information associated with the plurality of additional neuromuscular signals;

receiving, as one or more outputs from the tuned statistical model, additional pose and/or force estimates associated with an additional musculoskeletal representation of the user; and

providing, to the user interface, a visual representation of the additional musculoskeletal representation using, in part, the additional pose and/or force estimates.

4. The method of claim 1 , wherein the user interface is presented on a head-mounted display of an artificial reality device.

5. The method of claim 1 , further comprising prompting the user to select a choice from each of a plurality of binary choices associated with different rendered visual representations, and

wherein receiving the user input comprises receiving input associated with the plurality of binary choices.

6. The method of claim 5 , further comprising:

aggregating the received input associated with the plurality of binary choices, and

adjusting one or more parameters of the one or more statistical models based on the aggregated input.

7. The method of claim 1 , further comprising overlaying the visual representation with a second candidate visual representation, wherein the visual representation and second visual representation are rendered using different colors.

8. The method of claim 1 , further comprising displaying the visual representation and a second candidate visual representation at a same time.

9. The method of claim 1 , further comprising rendering the visual representation and a second candidate visual representation side-by-side.

10. The method of claim 1 , wherein the gesture comprises a continuous gesture, and providing the visual representation comprises generating at least one animation representing the gesture, wherein the at least one animation comprises the visual representation rendered at a plurality of points in time.

11. The method of claim 10 , wherein receiving the user input comprises receiving the user input during replay of at least a portion of the at least one animation.

12. The method of claim 10 , further comprising:

receiving a selection of one of the plurality of points in time of the animation, and

wherein receiving the user input comprises receiving the user input subsequent to receiving the selection of one of the plurality of points in time of the animation.

13. The method of claim 1 , wherein the musculoskeletal representation is a musculoskeletal representation associated with a hand, and

wherein the pose and/or force estimates comprise handstate estimates.

14. The method of claim 1 , wherein the user interface is a graphical user interface configured to visually instruct the user to perform the gesture.

15. The method of claim 1 , further comprising displaying a likelihood or quality metric associated with the output from the one or more statistical models.

16. An electronic device comprising:

a communications interface that is in communication with (i) one or more neuromuscular sensors configured to detect a plurality of neuromuscular signals from a user and (ii) a user interface;

a plurality of processors; and

memory storing one or more programs configured to be executed by the plurality of processors, the one or more programs including instructions for:

receiving, from the one or more neuromuscular sensors via the communications interface, a plurality of neuromuscular signals detected during performance of a gesture by a user;

providing, as input to one or more statistical models, information associated with the plurality of neuromuscular signals, wherein the one or more statistical models output pose and/or force estimates for the gesture;

providing, to the user interface via the communications interface, a visual musculoskeletal representation generated based on the pose and/or force estimates from the one or more statistical models;

receiving, from the user interface via the communications interface, user input to adjust the visual musculoskeletal representation to more closely conform to the gesture;

based on the user input, determining a tuned statistical model for the user that predicts the gesture more accurately than the one or more statistical models when using the information associated with the plurality of neuromuscular signals as input by selecting the tuned statistical model from the one or more statistical models; and

switching one or more times between displaying the visual representation and a second candidate visual representation prior to receiving the user input.

17. The electronic device of claim 16 , wherein determining the tuned statistical model comprises determining the statistical model that resulted in the adjusted visual representation.

18. The electronic device of claim 16 , wherein the one or more programs further include instructions for:

receiving, from the one or more neuromuscular sensors, a plurality of additional neuromuscular signals detected during performance of an additional gesture by the user;

providing, as input to the tuned statistical model, information associated with the plurality of additional neuromuscular signals;

receiving, as one or more outputs from the tuned statistical model, additional pose and/or force estimates associated with an additional musculoskeletal representation of the user; and

providing, to the user interface, a visual representation of the additional musculoskeletal representation using, in part, the additional pose and/or force estimates.

19. The electronic device of claim 16 , wherein the one or more programs further include instructions for repeatedly switching between displaying the visual representation and a second candidate visual representation prior to receiving the user input.

20. The electronic device of claim 16 , wherein the one or more programs further include instructions for prompting the user to select a choice from each of a plurality of binary choices associated with different rendered visual representations, and

wherein receiving the user input comprises receiving input associated with the plurality of binary choices.

21. The electronic device of claim 20 , wherein the one or more programs further include instructions for:

aggregating the received input associated with the plurality of binary choices, and

adjusting one or more parameters of the one or more statistical models based on the aggregated input.

22. The electronic device of claim 16 , wherein the one or more programs further include instructions for overlaying the visual representation with a second candidate visual representation, wherein the visual representation and second visual representation are rendered using different colors.

23. The electronic device of claim 16 , wherein the one or more programs further include instructions for displaying the visual representation and a second candidate visual representation at a same time.

24. The electronic device of claim 16 , wherein the one or more programs further include instructions for rendering the visual representation and a second candidate visual representation side-by-side.

25. The electronic device of claim 16 , wherein the gesture comprises a continuous gesture, and providing the visual representation comprises generating at least one animation representing the gesture, wherein the at least one animation comprises the visual representation rendered at a plurality of points in time.

26. The electronic device of claim 25 , wherein receiving the user input comprises receiving the user input during replay of at least a portion of the at least one animation.

27. The electronic device of claim 25 , wherein the one or more programs further include instructions for:

receiving a selection of one of the plurality of points in time of the animation, and

wherein receiving the user input comprises receiving the user input subsequent to receiving the selection of one of the plurality of points in time of the animation.

28. The electronic device of claim 16 , wherein the musculoskeletal representation is a musculoskeletal representation associated with a hand, and

wherein the pose and/or force estimates comprise handstate estimates.

29. An artificial-reality system comprising:

a head-mounted display; and

one or more electronic devices that are in communication with the head-mounted display, the one or more electronic devices being configured to:

receive a plurality of neuromuscular signals from a user during performance of a gesture while the user is wearing a first electronic device of the one or more electronic devices, wherein the first electronic device includes one or more neuromuscular sensors configured to detect the neuromuscular signals;

provide, as input to one or more statistical models, information associated with the plurality of neuromuscular signals;

wherein the head-mounted display is configured to:

receive, as output(s) from the one or more statistical models, pose and/or force estimates associated with a musculoskeletal representation of the user;

generate a visual representation of the musculoskeletal representation using, in part, the pose and/or force estimates;

process user input, based on the generated visual representation, to adjust the generated visual representation to more closely conform to the gesture;

based on the user input, determining a tuned statistical model for the user that predicts the gesture more accurately than the one or more statistical models when using the information associated with the plurality of neuromuscular signals as input by selecting the tuned statistical model from the one or more statistical models; and

switch one or more times between displaying the visual representation and a second candidate visual representation prior to receiving the user input.

30. The artificial-reality system of claim 29 , wherein:

the head-mounted display comprises a user interface; and

the user input is received via the user interface.

Assignments (4)
CHANGE OF NAME Recorded May 26, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060199/0876 →
CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY DATA PREVIOUSLY RECORDED AT REEL: 051649 FRAME: 0001. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Feb 10, 2020
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 051867/0136 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 20, 2020
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, INC.
Reel/Frame 051649/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 13, 2019
From: KAIFOSH, PATRICK; BERENZWEIG, ADAM; AL-NATSHEH, ADAM
To: CTRL-LABS CORPORATION
Reel/Frame 051288/0126 →