Navigating a user interface using in-air gestures detected via neuromuscular-signal sensors of a wearable device, and systems and methods of use thereof
The various implementations described herein include methods and systems for using hand gestures detected at a wearable device to navigate a user interface. An example method includes receiving, via one or more neuromuscular-signal sensors of a wrist-wearable device worn by a user, data generated during performance of an in-air wrist movement by a wrist of the user. The method also includes moving a point of focus on the user interface in accordance with the in-air wrist movement; and receiving, via the one or more neuromuscular-signal sensors, additional data generated during performance of an in-air gesture by the user. The method further includes determining that the in-air gesture is an execution gesture; and executing a command corresponding to the execution gesture.
1. A method, comprising:
causing display of a user interface comprising a plurality of control elements having a plurality of distinct locations on the user interface;
receiving, via one or more neuromuscular-signal sensors of a wrist-wearable device worn by a user, data generated during performance of an in-air gesture that includes movement by one or more of a wrist of the user, a hand of the user, and an arm of the user;
in response to receiving the data, causing display of a point of focus element to move independently of the plurality of control elements on the user interface in accordance with the in-air gesture such that the point of focus element moves from a first location to a location of a control element of the plurality of control elements, thereby selecting the control element of the user interface;
wherein causing the point of focus element to move comprises:
in accordance with a determination that a portion of the in-air gesture is a phalange-based gesture, before causing the point of focus element to move to the location of the control element, causing the point of focus element to snap from a first control element in the user interface to a second control element in the user interface; or
in accordance with a determination that a portion of the in-air gesture is a wrist-based or forearm-based gesture, before causing the point of focus element to move to the location of the control element, causing the point of focus element to move to a location distinct from the locations of the plurality of control elements in the user interface according to directionality of the in-air gesture;
while the point of focus element is selecting the control element of the user interface, receiving, via the one or more neuromuscular-signal sensors, additional data generated during performance of an additional in-air gesture by the user;
determining that the additional in-air gesture is an execution gesture; and
executing a command corresponding to the execution gesture and the control element.
2. The method of claim 1 , further comprising, prior to causing the point of focus element to move;
identifying the in-air gesture as being part of a navigation gesture, wherein the point of focus element is moved in accordance with the identification, or
identifying an initial position of the wrist of the user, wherein the point of focus element is moved in accordance with movement of the wrist of the user from the initial position.
3. The method of claim 1 , wherein the in-air gesture comprises the wrist of the user rotating from an initial position to a rotated position, and wherein the wrist of the user is maintained in the rotated position for an amount of time; and
the method further comprises, while the wrist of the user is maintained in the rotated position for the amount of time, causing the point of focus element to scroll through the plurality of control elements on the user interface.
4. The method of claim 3 , further comprising causing the point of focus element to cease to scroll and causing selection of a nearest control element to the point of focus element in accordance with the wrist of the user returning to the initial position.
5. The method of claim 1 , further comprising, in accordance with a determination that the point of focus element is within a threshold distance of a third control element on the user interface, causing selection of the third control element by causing the point of focus element to snap to the third control element.
6. The method of claim 5 , further comprising:
after selecting the third control element, detecting a user-interface control gesture; and
in accordance with a determination that movement of the user-interface control gesture would cause the point of focus element to move beyond a second threshold distance of the third control element, causing the third control element to be deselected.
7. The method of claim 6 , further comprising, in accordance with causing the third control element to be deselected, causing selection of a different control element by causing the point of focus element to snap to the different control element.
8. The method of claim 1 , wherein the point of focus element has a first appearance while selecting the control element and a second appearance while not selecting any control element of the user interface, the second appearance being different than the first appearance.
9. The method of claim 1 , wherein the point of focus element is caused to be displayed while a portion of the in-air gesture is maintained, and wherein the point of focus element ceases to be displayed in accordance with the portion of the in-air gesture being released.
10. A system comprising:
one or more processors; and
memory comprising instructions for:
causing display of a user interface comprising a plurality of control elements having a plurality of distinct locations on the user interface;
receiving, via one or more neuromuscular-signal sensors of a wrist-wearable device worn by a user, data generated during performance of an in-air gesture that includes movement by a wrist of the user;
in response to receiving the data, causing a display of a point of focus element to move independently of the plurality of control elements on the user interface in accordance with the in-air gesture such that the point of focus element moves from a first location to a location of a control element of the plurality of control elements, thereby selecting the control element of the user interface;
wherein causing the point of focus element to move comprises:
in accordance with a determination that a portion of the in-air gesture is a phalange-based gesture, before causing the point of focus element to move to the location of the control element, causing the point of focus element to snap from a first control element in the user interface to a second control element in the user interface; or
in accordance with a determination that a portion of the in-air gesture is a wrist-based or forearm-based gesture, before causing the point of focus element to move to the location of the control element, causing the point of focus element to move to a location distinct from locations of the plurality of control elements in the user interface according to directionality of the in-air gesture;
while the point of focus element is selecting the control element of the user interface, receiving, via the one or more neuromuscular-signal sensors, additional data generated during performance of an additional in-air gesture by the user;
determining that the additional in-air gesture is an execution gesture; and
executing a command corresponding to the execution gesture and the control element.
11. The system of claim 10 , wherein the memory further comprises instructions for, prior to causing the point of focus element to move;
identifying the in-air gesture as being part of a navigation gesture, wherein the point of focus element is moved in accordance with the identification, or
identifying an initial position of the wrist of the user, wherein the point of focus element is moved in accordance with movement of the wrist of the user from the initial position.
12. The system of claim 10 , wherein the in-air gesture comprises the wrist of the user rotating from an initial position to a rotated position, and wherein the wrist of the user is maintained in the rotated position for an amount of time; and
the memory further comprises instructions for, while the wrist of the user is maintained in the rotated position for the amount of time, causing the point of focus element to scroll through the plurality of control elements on the user interface.
13. The system of claim 12 , wherein the memory further comprises instructions for causing the point of focus element to cease to scroll and causing selection of a nearest control element to the point of focus element in accordance with the wrist of the user returning to the initial position.
14. The system of claim 10 , wherein the point of focus element has a first appearance while selecting the control element and a second appearance while not selecting any control element of the user interface, the second appearance being different than the first appearance.
15. The system of claim 10 , wherein the point of focus element is caused to be displayed while a portion of the in-air gesture is maintained, and wherein the point of focus element ceases to be displayed in accordance with the portion of the in-air gesture being released.
16. A non-transitory computer-readable storage medium including instructions that, when executed by a wearable device, cause the wearable device to:
cause display of a user interface comprising a plurality of control elements having a plurality of distinct locations on the user interface;
receive, via one or more neuromuscular-signal sensors of a wrist-wearable device worn by a user, data generated during performance of an in-air gesture that includes movement by one or more of a wrist of the user, a hand of the user, and an arm of the user;
in response to receiving the data, cause a display of a point of focus element to move independently of the plurality of control elements on the user interface in accordance with the in-air gesture such that the point of focus element moves from a first location to a location of a control element of the plurality of control elements, thereby selecting the control element of the user interface;
wherein causing the point of focus element to move comprises:
in accordance with a determination that a portion of the in-air gesture is a phalange-based gesture, before causing the point of focus element to move to the location of the control element, cause the point of focus element to snap from a first control element in the user interface to a second control element in the user interface; or
in accordance with a determination that a portion of the in-air gesture is a wrist-based or forearm-based gesture, before causing the point of focus element to move to the location of the control element, cause the point of focus element to move to a location distinct from the locations of the plurality of control elements in the user interface according to directionality of the in-air gesture;
while the point of focus element is selecting the control element of the user interface, receive, via the one or more neuromuscular-signal sensors, additional data generated during performance of an additional in-air gesture by the user;
determine that the additional in-air gesture is an execution gesture; and
execute a command corresponding to the execution gesture and the control element.
17. The non-transitory computer-readable storage medium of claim 16 , further comprising instructions that, when executed by the wearable device, cause the wearable device to, prior to causing the point of focus element to move;
identify the in-air gesture as being part of a navigation gesture, wherein the point of focus element is moved in accordance with the identification, or
identify an initial position of the wrist of the user, wherein the point of focus element is moved in accordance with movement of the wrist of the user from the initial position.
18. The non-transitory computer-readable storage medium of claim 16 , wherein the in-air gesture comprises the wrist of the user rotating from an initial position to a rotated position, and wherein the wrist of the user is maintained in the rotated position for an amount of time; and
further comprising instructions that, when executed by the wearable device, cause the wearable device to, while the wrist of the user is maintained in the rotated position for the amount of time, cause the point of focus element to scroll through the plurality of control elements on the user interface.
19. The non-transitory computer-readable storage medium of claim 16 , wherein the point of focus element has a first appearance while selecting the control element and a second appearance while not selecting any control element of the user interface, the second appearance being different than the first appearance.
20. The non-transitory computer-readable storage medium of claim 16 , wherein the point of focus element is caused to be displayed while a portion of the in-air gesture is maintained, and wherein the point of focus element ceases to be displayed in accordance with the portion of the in-air gesture being released.