Text input detection using neuromuscular-signal sensors of a wearable device, and systems and methods of use thereof
The various implementations described herein include techniques and apparatuses for handwriting and other user gesture detection and recognition. In one aspect, a method includes receiving, via one or more sensors of a wearable device worn by a user, neuromuscular data generated from performance of a hand gesture by the user. The method also includes generating, using one or more encoders, one or more encodings by encoding the neuromuscular data; and identifying, using a language model, one or more terms from the one or more encodings. The method further includes causing display of the one or more terms to the user during performance of the hand gesture.
1 . A method of gesture recognition, the method comprising:
receiving, via one or more sensors of a wearable device worn by a user, neuromuscular data generated from performance of a hand gesture by the user;
generating, using a temporal attention model, one or more encodings by encoding the neuromuscular data, wherein the temporal attention model uses at least one of a segmented attention mechanism and a sliding-window attention mechanism;
identifying, using one or more decoders, one or more terms from the one or more encodings; and
causing display of the one or more terms to the user during performance of the hand gesture.
2 . The method of claim 1 , wherein the neuromuscular data corresponds to two or more channels of neuromuscular data, and wherein each channel of the two or more channels corresponds to a distinct muscle activation.
3 . The method of claim 1 , wherein the temporal attention model comprises at least one of a conformer model and a streaming transformer model.
4 . The method of claim 1 , further comprising:
receiving, via one or more additional sensors, additional data generated from performance of the hand gesture;
generating one or more additional encodings by encoding the additional data; and
obtaining one or more combined encodings by combining the one or more encodings and the one or more additional encodings; and
wherein the one or more terms are identified by inputting the one or more combined encodings into a respective decoder of the one or more decoders.
5 . The method of claim 4 , wherein the one or more additional sensors comprise an inertial measurement unit (IMU) sensor.
6 . The method of claim 1 , wherein the hand gesture by the user corresponds to the user performing a handwriting gesture on a surface, and wherein the one or more sensors comprise an electromyography (EMG) sensor.
7 . The method of claim 6 , wherein the hand gesture is recognized in accordance with a determination that the user is holding a writing instrument or simulating holding a writing instrument.
8 . The non-transitory computer-readable storage medium of claim 7 , wherein the hand gesture is recognized in accordance with a determination that the user is holding a writing instrument or simulating holding a writing instrument.
9 . A non-transitory computer-readable storage medium including instructions which, when executed by a wearable device worn by a user, cause the wearable device to:
receive, via one or more sensors of the wearable device, neuromuscular data generated from performance of a hand gesture by the user;
generate, using a temporal attention model, one or more encodings by encoding the neuromuscular data, wherein the temporal attention model uses at least one of a segmented attention mechanism and a sliding window attention mechanism;
identify, using one or more decoders, one or more terms from the one or more encodings; and
cause display of the one or more terms to the user during performance of the hand gesture.
10 . The non-transitory, computer-readable storage medium of claim 9 , wherein the neuromuscular data corresponds to two or more channels of neuromuscular data, and wherein each channel of the two or more channels corresponds to a distinct muscle activation.
11 . The non-transitory computer-readable storage medium of claim 9 , wherein the temporal attention model comprises at least one of a conformer model and a streaming transformer model.
12 . The non-transitory, computer-readable storage medium of claim 9 , further comprising instructions for:
receiving, via one or more additional sensors, additional data generated from performance of the hand gesture;
generating one or more additional encodings by encoding the additional data; and
obtaining one or more combined encodings by combining the one or more encodings and the one or more additional encodings; and
wherein the one or more terms are identified by inputting the one or more combined encodings into a respective decoder of the one or more decoders.
13 . The non-transitory computer-readable storage medium of claim 12 , wherein the one or more additional sensors comprise an inertial measurement unit (IMU) sensor.
14 . The non-transitory computer-readable storage medium of claim 9 , wherein the hand gesture by the user corresponds to the user performing a handwriting gesture on a surface, and wherein the one or more sensors comprise an electromyography (EMG) sensor.
15 . A system, comprising:
memory storing one or more programs that, when executed by one or more processors of a wearable device, cause performance of:
receiving, via one or more sensors of the wearable device, neuromuscular data generated from performance of a hand gesture by a user;
generating, using a temporal attention model, one or more encodings by encoding the neuromuscular data, wherein the temporal attention model uses at least one of a segmented attention mechanism and a sliding window attention mechanism;
identifying, using one or more decoders, one or more terms from the one or more encodings; and
causing display of the one or more terms to the user during performance of the hand gesture.
16 . The system of claim 15 , wherein the neuromuscular data corresponds to two or more channels of neuromuscular data, and wherein each channel of the two or more channels corresponds to a distinct muscle activation.
17 . The system of claim 15 , wherein the hand gesture by the user corresponds to the user performing a handwriting gesture on a surface, and wherein the one or more sensors comprise an electromyography (EMG) sensor.
18 . The system of claim 17 , wherein the hand gesture is recognized in accordance with a determination that the user is holding a writing instrument or simulating holding a writing instrument.
19 . The system of claim 15 , wherein the one or more programs further cause performance of:
receiving, via one or more additional sensors, additional data generated from performance of the hand gesture;
generating one or more additional encodings by encoding the additional data; and
obtaining one or more combined encodings by combining the one or more encodings and the one or more additional encodings; and
wherein the one or more terms are identified by inputting the one or more combined encodings into a respective decoder of the one or more decoders.
20 . The system of claim 19 , wherein the one or more additional sensors comprise an inertial measurement unit (IMU) sensor.