Multi-stage gestures detected based on neuromuscular-signal sensors of a wearable device to activate user-interface interactions with low-false positive rates, and systems and methods of use thereof
The various implementations described herein include methods and systems for using a multi-stage in-air hand gesture to activate user-interface interactions in a way that ensures low-false positive rates. In one aspect, a method includes, while a gating gesture is maintained, receiving a first indication of performance of an adjustment gesture of a first magnitude directed to a user interface object associated with a plurality of values. The method further includes, in response to receiving the first indication, adjusting the user interface object to have a first state after moving through some of the plurality of values based on the first magnitude. The method also includes, after receiving an indication of a release of the gating gesture, in response to receiving a second indication of performance of the adjustment gesture, forgoing adjusting the user interface object such that the user interface object continues to have the first state.
1. A method of using multi-stage in-air hand gestures to perform interactions, the method comprising:
receiving, via one or more sensors of a wrist-wearable device worn by a user, data generated from performance of a multi-stage in-air hand gesture by the user;
in accordance with a determination that a first stage of the multi-stage in-air hand gesture is a priming gesture, activating a user interface and one or more additional sensors of the wrist-wearable device;
in accordance with a determination that a last stage of the multi-stage in-air hand gesture is a control gesture received within a predefined threshold amount of time after the priming gesture, executing a command for the user interface that corresponds to the control gesture, wherein the control gesture is detected using at least the one or more additional sensors;
deactivating the user interface and the one or more additional sensors;
after deactivating the user interface and the one or more additional sensors, receiving, via the one or more sensors of the wrist-wearable device worn by the user, additional data generated from performance of another multi-stage in-air hand gesture by the user;
in accordance with a determination that a first stage of the other multi-stage in-air hand gesture is another priming gesture, activating the user interface and the one or more additional sensors; and
in accordance with a determination that the control gesture is not detected within the predefined threshold amount of time of the other priming gesture, deactivating the user interface and the one or more additional sensors.
2. The method of claim 1 , wherein:
the priming gesture includes maintaining a pinch-and-hold gesture in which at least two phalanges of a hand of the user remain in contact for at least a preset amount of time; and
activating the user interface includes displaying a plurality of selectable user interface elements within the user interface while the pinch-and-hold gesture is maintained.
3. The method of claim 2 , further comprising ceasing to display the plurality of selectable user interface elements in accordance with a determination, based on data from the one or more sensors of the wrist-wearable device, that the pinch-and-hold gesture is no longer maintained.
4. The method of claim 2 , wherein:
the multi-stage in-air hand gesture further includes a navigation stage;
the navigation stage includes a rotation of a wrist of the user to navigate through a plurality of user interface elements in the user interface and select a selected user interface element from the plurality of user interface elements; and
the control gesture comprises a release of the pinch-and-hold gesture to execute a command for the selected user interface element.
5. The method of claim 2 , wherein:
the multi-stage in-air hand gesture further includes a navigation stage; and
the navigation stage includes a sliding movement of a first phalange of the at least two phalanges across a surface of a second phalange of the at least two phalanges, the first and second phalanges being located on different fingers of the hand of the user.
6. The method of claim 2 , wherein:
the pinch-and-hold gesture is maintained via a pressing force between the at least two phalanges of the hand of the user, the pressing force not meeting a preset activation criterion;
the control gesture comprises an increase in the pressing force between the at least two phalanges of the hand of the user, such that the pressing force becomes an increased pressing force; and
the command is executed in accordance with the increased pressing force meeting the preset activation criterion.
7. The method of claim 1 , further comprising identifying the command for the user interface based on a combination of the priming gesture and the control gesture.
8. The method of claim 1 , wherein activating the user interface includes presenting a user interface element that was not presented prior to activating the user interface.
9. The method of claim 1 , further comprising activating, in response to the priming gesture, a display of the wrist-wearable device or a head-wearable device.
10. The method of claim 1 , wherein activating the user interface includes causing a plurality of user interface elements to be displayed within the user interface.
11. The method of claim 10 , wherein:
activating the user interface includes causing the plurality of user interface elements to be displayed for a preset amount of time, and
the plurality of user interface elements cease to be displayed automatically, without further instructions from the user, when a control gesture is not detected within the predefined threshold amount of time.
12. The method of claim 10 , further comprising, in conjunction with executing the command, deactivating the user interface including ceasing to display the plurality of user interface elements.
13. The method of claim 1 , wherein:
the priming gesture comprises a double tap gesture in which at least one digit of a hand of the user contacts at least a part of the hand of the user twice in succession;
the control gesture comprises a snapping gesture in which at least two digits of a hand of the user touch and slide against one another; and
the command, which is identified based on the priming gesture and the control gesture, comprises capturing a video with an imaging device communicatively coupled to the wrist-wearable device.
14. The method of claim 1 , wherein activating the user interface includes causing one or more options to be presented to the user, the one or more options corresponding to available control gestures, including the control gesture that is the last stage of the multi-stage in-air hand gesture.
15. The method of claim 1 , further comprising:
after determining that the last stage of the multi-stage in-air hand gesture is the control gesture and prior to executing the command, causing a confirmation to be presented to the user, the confirmation requesting that the user confirm that execution of the command is intended by the user, wherein the command is executed in response to the user confirming that execution of the command is intended.
16. The method of claim 1 , wherein:
the priming gesture is performed while a first user interface element is in focus within the user interface, and
the method further includes:
in conjunction with activating the user interface, causing display of a plurality of options for interacting with the first user interface element, each of the plurality of options associated with a respective command for the user interface; and
in accordance with a determination that a second stage of the multi-stage in-air hand gesture, detected before the last stage of the multi-stage in-air hand gesture, is a navigation gesture, navigating through the plurality of options for interacting with the first user interface element until a respective option of the plurality of options is in focus within the user interface, the respective option of the plurality of options associated with the command, and
wherein the control gesture is performed while the respective option remains in focus within the user interface, thereby causing the executing of the command that is associated with the respective option.
17. The method of claim 1 , wherein the one or more sensors of the wrist-wearable device include an inertial measurement unit (IMU) sensor and the one or more additional sensors of the wrist-wearable device include an electromyography sensor.
18. The method of claim 1 , wherein the one or more sensors of the wrist-wearable device and the one or more additional sensors of the wrist-wearable device comprise neuromuscular sensors.
19. A system comprising:
one or more processors; and
memory coupled to the one or more processors, the memory comprising instructions for:
receiving, via one or more sensors of a wrist-wearable device worn by a user, data generated from performance of a multi-stage in-air hand gesture by the user;
in accordance with a determination that a first stage of the multi-stage in-air hand gesture is a priming gesture, activating a user interface and one or more additional sensors of the wrist-wearable device;
in accordance with a determination that a last stage of the multi-stage in-air hand gesture is a control gesture received within a predefined threshold amount of time after the priming gesture, executing a command for the user interface that corresponds to the control gesture, wherein the control gesture is detected using at least the one or more additional sensors;
deactivating the user interface and the one or more additional sensors;
after deactivating the user interface and the one or more additional sensors, receiving, via the one or more sensors of the wrist-wearable device worn by the user, additional data generated from performance of another multi-stage in-air hand gesture by the user;
in accordance with a determination that a first stage of the other multi-stage in-air hand gesture is another priming gesture, activating the user interface and the one or more additional sensors; and
in accordance with a determination that the control gesture is not detected within the predefined threshold amount of time of the other priming gesture, deactivating the user interface and the one or more additional sensors.
20. A non-transitory computer-readable storage medium including instructions that, when executed by a wearable device, cause the wearable device to:
receive, via one or more sensors of a wrist-wearable device worn by a user, data generated from performance of a multi-stage in-air hand gesture by the user;
in accordance with a determination that a first stage of the multi-stage in-air hand gesture is a priming gesture, activate a user interface and one or more additional sensors of the wrist-wearable device;
in accordance with a determination that a last stage of the multi-stage in-air hand gesture is a control gesture received within a predefined threshold amount of time after the priming gesture, execute a command for the user interface that corresponds to the control gesture, wherein the control gesture is detected using at least the one or more additional sensors;
deactivate the user interface and the one or more additional sensors;
after deactivating the user interface and the one or more additional sensors, receive, via the one or more sensors of the wrist-wearable device worn by the user, additional data generated from performance of another multi-stage in-air hand gesture by the user;
in accordance with a determination that a first stage of the other multi-stage in-air hand gesture is another priming gesture, activate the user interface and the one or more additional sensors; and
in accordance with a determination that the control gesture is not detected within the predefined threshold amount of time of the other priming gesture, deactivate the user interface and the one or more additional sensors.