Devices, systems, and methods for controlling computing devices via neuromuscular signals of users
The disclosed human computer interface (HCI) system may include (1) at least one processor, (2) a plurality of sensors that detect one or more neuromuscular signals from a forearm or wrist of a user, and (3) memory that stores (A) one or more trained inferential models that determine an amount of force associated with the one or more neuromuscular signals and (B) computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to (I) identify the amount of force determined by the one or more trained inferential models, (II) determine that the amount of force satisfies a threshold force value, and in accordance with the determination that the amount of force satisfies the threshold force value, (III) generate a first input command for the HCI system. Various other devices, systems, and methods are also disclosed.
1. A human-computer interface (HCI) system comprising:
at least one processor;
a plurality of sensors that detect one or more neuromuscular signals from a forearm or wrist of a user, wherein the plurality of sensors are arranged on one or more wearable devices; and
memory that stores:
one or more trained inferential models that:
determine an amount of force associated with the one or more neuromuscular signals detected by the plurality of sensors; and
determine one or more hand gestures performed by the user based at least in part on the one or more neuromuscular signals detected by the plurality of sensors;
computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to:
identify the amount of force associated with the one or more neuromuscular signals as determined by the one or more trained inferential models;
determine that the amount of force satisfies a threshold force value;
identify the one or more hand gestures performed by the user as determined by the one or more trained inferential models;
determine that the amount of force was exerted by the user while performing the one or more hand gestures;
maintain a mapping between a set of hand gestures and a set of actions capable of being performed by the HCI system;
maintain an additional mapping between the set of hand gestures and an additional set of actions capable of being performed by the HCI system;
activate the mapping;
select an action to be performed by the HCI system based at least in part on the mapping and the one or more hand gestures performed by the user due at least in part on the mapping having been activated;
generate a first input command for the HCI system to direct the HCI system to perform the action due at least in part to:
the amount of force satisfying the threshold force value; and
the amount of force being exerted by the user while performing the one or more hand gestures;
deactivate the mapping after generating the first input command for the HCI system;
activate the additional mapping; and
after activating the additional mapping, generate a second input command to direct the HCI system to perform a different action included in the additional mapping in accordance with a subsequent determination that the user performed the one or more hand gestures.
2. The HCI system of claim 1 , wherein:
the threshold force value is a first threshold force value; and
the computer-executable instructions, when executed by the at least one processor, further cause the at least one processor to:
determine whether the amount of force also satisfies a second threshold force value that is greater than the first threshold force value; and
in accordance with a determination that the amount of force satisfies the first threshold force value and the second threshold force value, generate another input command that differs from the first input command for the HCI system.
3. The HCI system of claim 1 , wherein:
the one or more trained inferential models determine an additional amount of force associated with the one or more neuromuscular signals detected by the plurality of sensors; and
the computer-executable instructions, when executed by the at least one processor, further cause the at least one processor to:
identify the additional amount of force associated with the one or more neuromuscular signals as determined by the one or more trained inferential models;
determine whether the additional amount of force does not satisfy the threshold force value; and
in accordance with a determination that the additional amount of force does not satisfy the threshold force value, forgo generation of an additional input command for the HCI system.
4. The HCI system of claim 1 , wherein:
the one or more hand gestures comprise a first hand gesture and a second hand gesture; and
the computer-executable instructions, when executed by the at least one processor, further cause the at least one processor to:
determine that the amount of force was exerted by the user while performing the first hand gesture;
determine whether an additional amount of force that satisfies an additional threshold force value was exerted by the user while performing the second hand gesture;
generate the first input command for the HCI system in accordance with a determination that the amount of force was exerted by the user while performing the first hand gesture; and
generate another input command for the HCI system in accordance with a determination that the additional amount of force that satisfies the additional threshold force value was exerted by the user while performing the second hand gesture.
5. The HCI system of claim 1 , wherein the computer-executable instructions, when executed by the at least one processor, further cause the at least one processor to:
map the action to be performed by the HCI system to a plurality of conditions;
determine, based at least in part on the one or more neuromuscular signals detected by the plurality of sensors, that the plurality of conditions mapped to the action have been satisfied; and
generate the first input command for the HCI system due at least in part to the plurality of conditions having been satisfied.
6. The HCI system of claim 1 , wherein the computer-executable instructions, when executed by the at least one processor, further cause the at least one processor to:
identify photographic data representative of the one or more hand gestures as captured by a camera; and
provide the photographic data to the one or more trained inferential models to enable the one or more trained inferential models to determine the one or more hand gestures based at least in part on the one or more neuromuscular signals and the photographic data.
7. The HCI system of claim 1 , wherein:
the one or more trained inferential models further determine a single hand gesture performed by the user based at least in part on the one or more neuromuscular signals detected by the plurality of sensors; and
the computer-executable instructions, when executed by the at least one processor, further cause the at least one processor to:
identify the single hand gesture performed by the user as determined by the one or more trained inferential models;
determine whether the amount of force and an additional amount of force were both exerted by the user while performing the single hand gesture;
generate the first input command for the HCI system in accordance with a determination that the amount of force was exerted by the user while performing the single hand gesture; and
generate another command for the HCI system in accordance with a determination that the additional amount of force was exerted by the user while performing the single hand gesture.
8. The HCI system of claim 1 , wherein:
the one or more trained inferential models determine an additional amount of force associated with the one or more neuromuscular signals detected by the plurality of sensors; and
the computer-executable instructions, when executed by the at least one processor, further cause the at least one processor to:
identify the additional amount of force associated with the one or more neuromuscular signals as determined by the one or more trained inferential models;
determine whether the additional amount of force does not satisfy the threshold force value; and
in accordance with a determination that the additional amount of force does not satisfy the threshold force value, generate another command that differs from the first input command for the HCI system.
9. The HCI system of claim 6 , wherein:
the one or more hand gestures comprise either:
an index finger pinch; or
a middle finger pinch; and
the one or more trained inferential models are able to distinguish between an index finger pinch and a middle finger pinch based at least in part on the one or more neuromuscular signals and the photographic data.
10. The HCI system of claim 7 , wherein:
the single hand gesture comprises a first formed by a hand of the user; and
the computer-executable instructions, when executed by the at least one processor, further cause the at least one processor to:
formulate the first input command to control a speed of a cursor implemented on the HCI system in accordance with the amount of force exerted by the user while performing the single hand gesture; and
formulate the first input command to increase or decrease the speed of the cursor implemented on the HCI system in accordance with the additional amount of force exerted by the user while performing the single hand gesture.
11. The HCI system of claim 8 , wherein the first input command or the second input command directs the HCI system to perform at least one of:
navigate a graphical user interface (GUI) of an application;
draw a visual element on a GUI of an application;
control a scrolling speed of a GUI of an application;
scroll through a GUI of an application;
modify a width of a virtual drawing instrument used to draw in a GUI of an application;
associate a cursor with a visual element displayed in a GUI of an application;
provide a feedback indication of an association made between a cursor and a visual element displayed in a GUI of an application;
move a selection indicator forward or backward along a radial menu displayed in a GUI of an application;
modify a characteristic of a visual element displayed in a GUI of an application;
display a visual element in a GUI of an application;
click a visual element on a GUI of an application; or
select a visual element on a GUI of an application.
12. The HCI system of claim 1 , wherein the one or more wearable devices, the at least one processor, and the memory are incorporated into an artificial-reality system that includes a head-mounted display.
13. An artificial-reality system comprising:
a head-mounted display; and
a human-computer interface (HCI) system communicatively coupled to the head-mounted display, wherein the HCI system comprises:
at least one processor;
a plurality of sensors that detect one or more neuromuscular signals from a forearm or wrist of a user, wherein the plurality of sensors are arranged on one or more wearable devices; and
memory that stores:
one or more trained inferential models that:
determine an amount of force associated with the one or more neuromuscular signals detected by the plurality of sensors; and
determine one or more hand gestures performed by the user based at least in part on the one or more neuromuscular signals detected by the plurality of sensors;
computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to:
identify the amount of force associated with the one or more neuromuscular signals as determined by the one or more trained inferential models;
determine that the amount of force satisfies a threshold force value;
identify the one or more hand gestures performed by the user as determined by the one or more trained inferential models;
determine that the amount of force was exerted by the user while performing the one or more hand gestures;
maintain a mapping between a set of hand gestures and a set of actions capable of being performed by the HCI system;
maintain an additional mapping between the set of hand gestures and an additional set of actions capable of being performed by the HCI system;
activate the mapping;
select an action to be performed by the HCI system based at least in part on the mapping and the one or more hand gestures performed by the user due at least in part on the mapping having been activated;
generate a first input command for the artificial-reality system to perform the action due at least in part to:
the amount of force satisfying the threshold force value; and
the amount of force being exerted by the user while performing the one or more hand gestures;
deactivate the mapping after generating the first input command for the HCI system;
activate the additional mapping; and
after activating the additional mapping, generate a second input command to direct the artificial-reality system to perform a different action included in the additional mapping in accordance with a subsequent determination that the user performed the one or more hand gestures.
14. The artificial-reality system of claim 13 , wherein:
the threshold force value is a first threshold force value; and
the computer-executable instructions, when executed by the at least one processor, further cause the at least one processor to:
determine whether the amount of force also satisfies a second threshold force value that is greater than the first threshold force value; and
in accordance with a determination that the amount of force satisfies the first threshold force value and the second threshold force value, generate another input command that differs from the first input command for the artificial-reality system.
15. The artificial-reality system of claim 14 , wherein:
the one or more trained inferential models determine an additional amount of force associated with the one or more neuromuscular signals detected by the plurality of sensors; and
the computer-executable instructions, when executed by the at least one processor, further cause the at least one processor to:
identify the additional amount of force associated with the one or more neuromuscular signals as determined by the one or more trained inferential models;
determine whether the additional amount of force does not satisfy the first threshold force value; and
in accordance with a determination that the additional amount of force does not satisfy the first threshold force value, forgo generation of an additional input command for the artificial-reality system.
16. A method comprising:
detecting, by a plurality of sensors incorporated into one or more wearable devices of a human-computer interface (HCI) system, one or more neuromuscular signals from a forearm or wrist of a user;
determining, by one or more trained inferential models implemented by at least one processor of the HCI system, an amount of force associated with the one or more neuromuscular signals detected by the plurality of sensors;
determining, by the one or more trained inferential models implemented by the at least one processor of the HCI system, one or more hand gestures performed by the user based at least in part on the one or more neuromuscular signals detected by the plurality of sensors;
determining, by the at least one processor of the HCI system, that the amount of force associated with the one or more neuromuscular signals satisfies a threshold force value;
identifying, by the at least one processor of the HCI system, the one or more hand gestures performed by the user as determined by the one or more trained inferential models;
determining, by the at least one processor of the HCI system, that the amount of force was exerted by the user while performing the one or more hand gestures;
maintaining, by the at least one processor of the HCI system, a mapping between a set of hand gestures and a set of actions capable of being performed by the HCI system;
maintaining, by the at least one processor of the HCI system, an additional mapping between the set of hand gestures and an additional set of actions capable of being performed by the HCI system;
activating, by the at least one processor of the HCI system, the mapping;
selecting, by the at least one processor of the HCI system, an action to be performed by the HCI system based at least in part on the mapping and the one or more hand gestures performed by the user due at least in part on the mapping having been activated;
generating, by the at least one processor of the HCI system, a first input command for the HCI system to direct the HCI system to perform the action due at least in part to:
the amount of force satisfying the threshold force value; and
the amount of force being exerted by the user while performing the one or more hand gestures;
deactivating, by the at least one processor of the HCI system, the mapping after generating the first input command for the HCI system;
activating, by the at least one processor of the HCI system, the additional mapping; and
generating, by the at least one processor of the HCI system after activating the additional mapping, a second input command to direct the HCI system to perform a different action included in the additional mapping in accordance with a subsequent determination that the user performed the one or more hand gestures.