IP Library Granted Patent US 11,829,519
Granted Patent B1
US 11,829,519 · App. 16/805,589 · Granted Nov 28, 2023

Systems, methods, and apparatuses for a wearable control device to facilitate performance of manufacturing and various motor tasks

Inventor: Anthony D. Moschella (Booklyn, NY)
Assignee: Meta Platforms Technologies, LLC
G06F3/011G05B19/0425G06F1/163G06V20/20
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,829,519
App. No.
16/805,589
Granted
Nov 28, 2023
Kind
B1
Abstract

Methods, systems, and apparatuses for providing feedback to a human operator during performance of a manufacturing or other motor task. The system comprises a plurality of neuromuscular sensors configured to collect or record a plurality of neuromuscular signals from the human operator, wherein the plurality of neuromuscular sensors are arranged on one or more wearable devices, and at least one computer processor programmed to determine, based on the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals, one or more metrics associated with a performance of the manufacturing or motor task by the human operator, and provide via an extended reality interface such as an augmented reality (AR) interface, feedback to the human operator regarding their performance of the manufacturing task, wherein the feedback is based, at least in part, on the determined one or more metrics.

Claims (61)

1. A system for providing feedback to a human operator, the system comprising:

a wearable structure configured to be worn on a wrist of the human operator, the wearable structure including an inner surface;

a plurality of neuromuscular sensors arranged on the inner surface of the wearable structure and configured to record a plurality of neuromuscular signals from the wrist of the human operator; and

at least one computer processor programmed to:

determine, based on the plurality of neuromuscular signals from the wrist of the human operator and/or information based on the plurality of neuromuscular signals from the wrist of the human operator, one or more metrics associated with a performance, by the human operator, of an ordered sequence of movements for one or more motor tasks involving a real-world object, the one or more metrics comprising one or more of:

a force applied by the human operator to the real-world object during performance of the ordered sequence of movements for the one or more motor tasks involving the real-world object; and

an indication of completion of one or more movements in the ordered sequence of movements for the one or more motor tasks involving the real-world object; and

provide, via an augmented reality (AR) interface, feedback to the human operator regarding their performance of the ordered sequence of movements for the one or more motor tasks involving the real-world object, the feedback comprising one or more of:

an indication of a difference between the force applied by the human operator to the real-world object and one or more target forces; and

an indication of a movement in the ordered sequence of movements for the one or more motor tasks having not been fully executed.

2. The system of claim 1 , wherein the at least one computer processor is further programmed to:

provide as input to a trained statistical model, the plurality of neuromuscular signals from the wrist of the human operator and/or information based on the plurality of neuromuscular signals from the wrist of the human operator; and

determine, based on an output of the trained statistical model, the one or more metrics associated with the performance of the ordered sequence of movements for the one or more motor tasks involving the real-world object.

3. The system of claim 1 , wherein:

the one or more metrics include information about the force applied by the human operator to the real-world object during performance of the ordered sequence of movements for the one or more motor tasks involving the real-world object; and

the feedback comprises the indication of the difference between the force applied by the human operator to the real-world object and the one or more target forces.

4. The system of claim 1 , wherein:

the one or more metrics include the indication of completion of the one or more movements in the ordered sequence of movements for the one or more motor tasks involving the real-world object; and

the feedback comprises the indication of the movement in the ordered sequence of movements for the one or more motor tasks having not been fully executed.

5. The system of claim 1 , wherein the one or more metrics further include information associated with a position of at least a portion of the human operator relative to the real-world object during performance of the ordered sequence of movements for the one or more motor tasks involving the real-world object.

6. The system of claim 5 , wherein the at least one computer processor is further programmed to monitor an angle of the at least a portion of the human operator relative to the real-world object during performance of the ordered sequence of movements for the one or more motor tasks involving the real-world object.

7. The system of claim 5 , wherein the at least one computer processor is further programmed to monitor at least six degrees of freedom of an arm and/or hand of the human operator during performance of the ordered sequence of movements for the one or more motor tasks involving the real-world object.

8. The system of claim 1 , wherein the one or more metrics associated with performance of the ordered sequence of movements for the one or more motor tasks involving the real-world object include information associated with a posture of the human operator during performance of the ordered sequence of movements for the one or more motor tasks involving the real-world object.

9. The system of claim 1 , wherein providing the feedback to the human operator via the AR interface comprises:

instructing an AR device in communication with the at least one computer processor to provide the feedback to the human operator in an augmented reality environment generated by the AR device.

10. The system of claim 1 , wherein the one or more motor tasks comprise at least one of one or more manufacturing tasks, one or more surgical tasks, or one or more military-related tasks involving the real-world object.

11. The system of claim 1 , wherein providing feedback to the human operator comprises:

displaying one or more visual prompts to the human operator via the AR interface, the one or more visual prompts including one or more instructions regarding the performance of the ordered sequence of movements.

12. The system of claim 1 , wherein providing feedback to the human operator comprises:

prompting the human operator to change an amount of force applied during performance of the ordered sequence of movements.

13. The system of claim 1 , wherein providing feedback to the human operator comprises:

providing one or more instructions to the human operator regarding an additional amount of force to be applied to the real-world object during performance of the ordered sequence of movements.

14. The system of claim 1 , wherein providing feedback to the human operator comprises:

providing a notification to the human operator indicative of whether the ordered sequence of movements was completed properly by the human operator.

15. The system of claim 14 , wherein providing feedback to the human operator further comprises, when it is determined that the ordered sequence of movements was not completed properly, providing, via the AR interface, instructions describing how to complete the ordered sequence of movements properly.

16. A system for training a human operator to perform one or more manufacturing tasks involving a real-world object, the system comprising:

a wearable device comprising:

a wearable structure configured to be worn on a wrist of the human operator, the wearable structure including an inner surface; and

a plurality of neuromuscular sensors arranged on the inner surface of the wearable structure and configured to record a plurality of neuromuscular signals from the wrist of the human operator;

an augmented reality (AR) system or a virtual reality (VR) system; and

at least one computer processor in communication with the plurality of neuromuscular sensors and the AR or VR system, wherein the at least one computer processor is programmed to:

instruct the AR or VR system to provide at least one prompt to the human operator about proper performance of each movement in an ordered sequence of movements for the one or more manufacturing tasks involving the real-world object;

determine, based on the plurality of neuromuscular signals from the wrist of the human operator and/or information based on the plurality of neuromuscular signals from the wrist of the human operator, one or more metrics associated with a performance, by the human operator, of the ordered sequence of movements for the one or more manufacturing tasks involving the real-world object, the one or more metrics comprising one or more of:

a force applied by the human operator to the real-world object during performance of the ordered sequence of movements for the one or more manufacturing tasks involving the real-world object; and

an indication of completion of one or more movements in the ordered sequence of movements for the one or more manufacturing tasks involving the real-world object; and

provide feedback to the human operator regarding whether the human operator has properly performed the ordered sequence of movements for the one or more manufacturing tasks involving the real-world object, the feedback comprising one or more of:

an indication of a difference between the force applied by the human operator to the real-world object and one or more target forces; and

an indication of a movement in the ordered sequence of movements for the one or more manufacturing tasks having not been fully executed.

17. The system of claim 16 , wherein providing feedback to the human operator comprises providing feedback via the AR or VR system during performance of the ordered sequence of movements for the one or more manufacturing tasks.

18. The system of claim 16 , wherein the at least one computer processor is programmed to monitor based, at least in part, on the recorded neuromuscular signals from the wrist of the human operator, the force applied to the real-world object by the human operator during performance of the ordered sequence of movements for the one or more manufacturing tasks, and

wherein providing feedback to the human operator comprises providing information describing the monitored force.

19. The system of claim 16 , wherein the at least one computer processor is further programmed to monitor based, at least in part, on the recorded neuromuscular signals from the wrist of the human operator, positions of two or more body parts of the human operator relative to the real-world object during performance of the ordered sequence of movements for the one or more manufacturing tasks, and

wherein providing feedback to the human operator further comprises providing information describing the monitored positions.

20. A method for providing feedback to a human operator during performance of one or more manufacturing tasks involving a real-world object, the method comprising:

receiving, via a plurality of neuromuscular sensors arranged on an inner surface of a wearable structure configured to be worn on a wrist of the human operator, a plurality of neuromuscular signals from the wrist of the human operator;

determining, based on the plurality of neuromuscular signals from the wrist of the human operator and/or information based on the plurality of neuromuscular signals from the wrist of the human operator, one or more metrics associated with a performance of an ordered sequence of movements for the one or more manufacturing tasks by the human operator, the one or more metrics comprising one or more of:

a force applied by the human operator to the real-world object during performance of the ordered sequence of movements for the one or more manufacturing tasks involving the real-world object; and

an indication of completion of one or more movements in the ordered sequence of movements for the one or more manufacturing tasks involving the real-world object; and

providing via an augmented reality (AR) interface, feedback to the human operator regarding their performance of the sequence of movements for the one or more manufacturing tasks, the feedback comprising one or more of:

an indication of a difference between the force applied by the human operator to the real-world object and one or more target forces; and

an indication of a movement in the ordered sequence of movements for the one or more manufacturing tasks having not been fully executed.

Assignments (2)
CHANGE OF NAME Recorded May 27, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060203/0228 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 11, 2020
From: MOSCHELLA, ANTHONY D.
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 052089/0140 →