Camera-guided interpretation of neuromuscular signals

1. A computerized system for providing dynamically-updated musculoskeletal information, the system comprising:

one or more neuromuscular sensors configured to obtain one or more neuromuscular signals produced by a user;

at least one camera configured to capture one or more contemporaneous images with the one or more neuromuscular signals; and

at least one computer processor programmed to:

derive contemporaneous contextual information from the one or more contemporaneous images, the contemporaneous contextual information comprising at least one of:

an indication of at least one musculoskeletal segment of the user exerting a force against at least one other musculoskeletal segment of the user when the one or more neuromuscular signals were produced by the user;

an indication of the least one musculoskeletal segment of the user exerting a force against an object when the one or more neuromuscular signals were produced by the user; or

an indication of an environment in which the user produced the one or more neuromuscular signals;

determine, based on the contemporaneous contextual information and the one or more neuromuscular signals, at least one of:

position information describing a spatial relationship between two or more connected musculoskeletal segments of the user;

first force information describing the force exerted by the at least one musculoskeletal segment of the user against the object; or

second force information describing the force exerted by the at least one musculoskeletal segment of the user against the at least one other musculoskeletal segment of the user; and

output at least one of the position information, the first force information, or the second force information.

2. The computerized system of claim 1 , wherein the at least one computer processor is further programmed to:

determine a first quality of the one or more neuromuscular signals, or a second quality of the one or more contemporaneous images, or both the first quality and the second quality; and

weight at least one of the one or more neuromuscular signals based on the first quality or the one or more contemporaneous images based on the second quality to determine the position information, the first force information, or the second force information.

3. The computerized system of claim 2 , wherein the first quality of the one or more neuromuscular signals is determined by determining whether at least one of the one or more neuromuscular signals includes at least one signal artifact.

4. The computerized system of claim 1 , wherein:

the two or more connected musculoskeletal segments include a forearm rigid segment connected via a wrist joint to a plurality of rigid segments for a hand;

the one or more neuromuscular sensors include one or more electromyography (EMG) sensors; and

the at least one computer processor is further programmed to:

determine the position information for at least one of the forearm rigid segment, the wrist joint, or the plurality of rigid segments in the hand based on the one or more neuromuscular signals output from the one or more EMG sensors and the one or more contemporaneous images captured by the at least one camera.

5. The computerized system of claim 1 , wherein:

a wrist of the user, a hand connected to the wrist, and fingers of the hand are captured by the one or more contemporaneous images; and

the two or more connected musculoskeletal segments comprise segments from the wrist, the hand, and the fingers of the user.

6. The computerized system of claim 5 , wherein:

the at least one camera captures the one or more contemporaneous images while the one or more neuromuscular signals are simultaneously obtained; and

the position information relates to any one or any combination of: a muscular activation state, a forearm or hand motion, a handstate, a posture, a dynamic gesture, a static gesture, and a pose.

7. The computerized system of claim 4 , further comprising at least one inertial measurement unit (IMU) sensor, wherein the at least one computer processor is further programmed to determine the position information for at least one of the forearm rigid segment, the wrist joint, and/or the plurality of rigid segments based on IMU signals output from the at least one IMU sensor.

8. The computerized system of claim 7 , wherein the at least one computer processor is further programmed to:

determine an initial position of at least one of the forearm rigid segment, the wrist joint, or the plurality of rigid segments based on the IMU signals output from the at least one IMU sensor; and

adjust the initial position for at least one of the forearm rigid segment, the wrist joint, or the plurality of rigid segments based on the one or more contemporaneous images captured by the at least one camera.

9. The computerized system of claim 1 , wherein:

the position information is determined based on the one or more contemporaneous images; and

the first force information or the second force information is determined based on the one or more neuromuscular signals.

10. The computerized system of claim 1 , wherein:

the contemporaneous contextual information comprises the indication of the environment in which the user produced the one or more neuromuscular signals; and

the indication of the environment is used to limit probabilities of spatial relationships between the two or more connected musculoskeletal segments of the user.

11. The computerized system of claim 9 , wherein:

the position information is determined by determining, based on the contemporaneous contextual information, that at least two of the two or more connected musculoskeletal segments of the user are touching; and

the second force information is determined by determining, based on the one or more neuromuscular signals, a force applied between the at least two of the two or more connected musculoskeletal segments.

12. The computerized system of claim 1 , wherein the one or more contemporaneous images includes any one or any combination of:

an image produced by visible light;

an image produced by infrared light;

an image produced by light of a predetermined range of wavelengths; and

an image produced by light of two or more different predetermined ranges of wavelengths.

13. The computerized system of claim 1 , wherein the contemporaneous contextual information comprises the indication of the at least one musculoskeletal segment of the user exerting the force against the at least one other musculoskeletal segment of the user.

14. The computerized system of claim 1 , wherein the contemporaneous contextual information comprises the indication of the least one musculoskeletal segment of the user exerting the force against the object.

15. A method for providing dynamically-updated musculoskeletal information, the method comprising:

receiving one or more neuromuscular signals obtained by one or more neuromuscular sensors from a user;

receiving, by at least one processor, one or more contemporaneous images captured by at least one camera when the one or more neuromuscular signals were produced by the user;

deriving contemporaneous contextual information from the one or more contemporaneous images, the contemporaneous contextual information comprising at least one of:

an indication of at least one musculoskeletal segment of the user exerting a force against at least one other musculoskeletal segment of the user when the one or more neuromuscular signals were produced by the user;

an indication of the least one musculoskeletal segment of the user exerting a force against an object when the one or more neuromuscular signals were produced by the user; or

an indication of an environment in which the user produced the one or more neuromuscular signals;

determining, based on the contemporaneous contextual information and the one or more neuromuscular signals, at least one of:

position information describing a spatial relationship between two or more connected musculoskeletal segments of the user;

first force information describing the force exerted by the at least one musculoskeletal segment of the user against the object; or

second force information describing the force exerted by the at least one musculoskeletal segment of the user against the at least one other musculoskeletal segment of the user; and

outputting, by the at least one processor, at least one of the position information, the first force information, or the second force information.

16. The method of claim 15 , further comprising:

determining, by the at least one processor, a first quality of the one or more neuromuscular signals, or a second quality of the one or more contemporaneous images, or both the first quality and the second quality; and

weighting, by the at least one processor, at least one of the one or more neuromuscular signals based on the first quality or the one or more contemporaneous images based on the second quality to determine the position information, the first force information, or the second force information.

17. The method of claim 16 , wherein the first quality of the one or more neuromuscular signals is determined by determining whether at least one of the one or more neuromuscular signals includes at least one signal artifact.

18. The method of claim 15 , wherein:

the two or more connected musculoskeletal segments include a forearm rigid segment connected via a wrist joint to a plurality of rigid segments for a hand;

the one or more neuromuscular sensors include one or more electromyography (EMG) sensors; and

the method further comprises:

determining, based on the one or more neuromuscular signals output from the one or more EMG sensors and the one or more contemporaneous images captured by the at least one camera, the position information for one or more of the following: the forearm rigid segment, the wrist joint, and the plurality of rigid segments for the hand.

19. The method of claim 15 , wherein:

a wrist of the user, a hand connected to the wrist, and fingers of the hand are captured by the one or more contemporaneous images; and

the two or more connected musculoskeletal segments comprise segments from the wrist, the hand, and the fingers of the user.

20. The method of claim 19 , wherein:

the at least one camera captures the one or more contemporaneous images while the one or more neuromuscular signals are simultaneously obtained; and

the position information relates to any one or any combination of: a muscular activation state, a forearm or hand motion, a handstate, a posture, a dynamic gesture, a static gesture, and a pose.

21. The method of claim 18 , further comprising:

receiving, by the at least one processor, inertial measurement unit (IMU) signals obtained by at least one IMU sensor; and

determining, by the at least one processor, the position information for at least one of the forearm rigid segment, the wrist joint, or the plurality of rigid segments for the hand based on the IMU signals.

22. The method of claim 21 , further comprising:

determining, by the at least one processor, an initial position of at least one of the forearm rigid segment, the wrist joint, or the plurality of rigid segments for the hand based on the IMU signals; and

adjusting the initial position for at least one of the forearm rigid segment, the wrist joint, or the plurality of rigid segments for the hand based on the one or more contemporaneous images captured by the at least one camera.

23. The method of claim 15 , wherein:

the position information is determined based on the one or more contemporaneous images; and

the first force information or the second force information is determined based on the one or more neuromuscular signals.

24. The method of claim 15 , wherein:

the contemporaneous contextual information comprises the indication of the environment in which the user produced the one or more neuromuscular signals; and

the indication of the environment in which the user produced the one or more neuromuscular signals is used to limit probabilities of spatial relationship between the two or more connected musculoskeletal segments of the user.

25. The method of claim 23 , wherein:

determining the position information comprises determining, based on the one or more contemporaneous images, that at least two of the two or more connected musculoskeletal segments of the user are touching; and

determining the second force information by determining, based on the one or more neuromuscular signals, a force applied between the at least two of the two or more connected musculoskeletal segments.

26. The method of claim 15 , wherein the one or more contemporaneous images includes any one or any combination of:

an image produced by visible light;

an image produced by infrared light;

an image produced by light of a predetermined range of wavelengths; and

an image produced by light of two or more different predetermined ranges of wavelengths.

27. The method of claim 15 , wherein the contemporaneous contextual information comprises the indication of the at least one musculoskeletal segment of the user exerting the force against the at least one other musculoskeletal segment of the user.

28. The method of claim 15 , wherein the contemporaneous contextual information comprises the indication of the least one musculoskeletal segment of the user exerting the force against the object.

29. A non-transitory computer-readable storage medium storing code that, when executed by at least one computer, causes the at least one computer to perform a method for providing dynamically-updated musculoskeletal information, wherein the method comprises:

receiving one or more neuromuscular signals obtained by one or more neuromuscular sensors from a user;

receiving, contemporaneously with the one or more neuromuscular signals, one or more contemporaneous images captured by at least one camera;

deriving contemporaneous contextual information from the one or more contemporaneous images, the contemporaneous contextual information comprising at least one of:

an indication of at least one musculoskeletal segment of the user exerting a force against at least one other musculoskeletal segment of the user when the one or more neuromuscular signals were produced by the user;

an indication of the least one musculoskeletal segment of the user exerting a force against an object when the one or more neuromuscular signals were produced by the user; or

an indication of an environment in which the user produced the one or more neuromuscular signals;

determining, based on the contemporaneous contextual information and the one or more neuromuscular signals, at least one of:

position information describing a spatial relationship between two or more connected musculoskeletal segments of the user;

first force information describing the force exerted by the at least one musculoskeletal segment of the user against the object; or

second force information describing the force exerted by the at least one musculoskeletal segment of the user against the at least one other musculoskeletal segment of the user; and

outputting at least one of the position information, the first force information, or the second force information.

30. The storage medium of claim 29 , wherein:

a wrist of the user, a hand connected to the wrist, and fingers of the hand are captured by the one or more contemporaneous images; and

the two or more connected musculoskeletal segments include at least one of the wrist, the hand, or the fingers of the user.