Low-power hand-tracking system for wearable device
A method for a low-power hand-tracking system is described. In one aspect, a method includes polling a proximity sensor of a wearable device to detect a proximity event, the wearable device includes a low-power processor and a high-power processor, in response to detecting the proximity event, operating a low-power hand-tracking application on the low-power processor based on proximity data from the proximity sensor, and ending an operation of the low-power hand-tracking application in response to at least one of: detecting and recognizing a gesture based on the proximity data, detecting without recognizing the gesture based on the proximity data, or detecting a lack of activity from the proximity sensor within a timeout period based on the proximity data.
1. A method comprising:
polling a proximity sensor of a wearable device to detect a proximity event, the wearable device comprising a low-power processor and a high-power processor;
in response to detecting the proximity event, performing a gesture detection and recognition routine using a low-power gesture recognition application executed on the low-power processor based on proximity data from the proximity sensor;
determining that the gesture detection and recognition routine executed on the low-power processor fails to recognize a gesture of a user of the wearable device; and
in response to determining that the gesture detection and recognition routine executed on the low-power processor fails to recognize the gesture activating a high-power gesture recognition application executed on the high-power processor.
2. The method of claim 1 , further comprising:
generating a level of confidence of the gesture based on a gesture dictionary, using a gesture recognition algorithm operating on the low-power processor; and
identifying the gesture in response to the level of confidence exceeding a preset threshold.
3. The method of claim 2 , further comprising:
identifying an operation corresponding to the gesture; and
requesting one of the low-power processor or the high-power processor to perform the operation.
4. The method of claim 2 , further comprising:
identifying an operation of a mixed reality application corresponding to the gesture; and
requesting the high-power processor to perform the operation of the mixed reality application.
5. The method of claim 1 , further comprising:
returning the wearable device to an idle state that polls the proximity sensor at a regular interval after the high-power gesture recognition application recognizes the gesture.
6. The method of claim 1 , further comprising:
identifying a preset number of gesture recognition tries corresponding to the low-power gesture recognition application, the preset number of gesture recognition tries comprising at least two tries;
determining that the gesture detection and recognition routine executed on the low-power processor fails to recognize the gesture after a number of tries exceeding the preset number of gesture recognition tries,
wherein activating the high-power gesture recognition application executed on the high-power processor is in response to determining that the gesture detection and recognition routine executed on the low-power processor fails to recognize the gesture after the number of tries exceeding the preset number of gesture recognition tries.
7. The method of claim 1 , wherein the low-power processor comprises a microcontroller, wherein the high-power processor comprises a system-on-chip (SoC), wherein the proximity sensor comprises a time-of-flight sensor.
8. The method of claim 1 , wherein the low-power processor is configured to only operate the proximity sensor and the low-power gesture recognition application, wherein the high-power processor is configured to operate all sensors of the wearable device.
9. The method of claim 1 , wherein the low-power hand tracking gesture recognition application uses a neural network to detect and recognize a hand gesture of a user of the wearable device based on the proximity data, the neural network being configured to recognize a first set of hand gestures, and
wherein the high-power processor comprises a high-power hand-tracking application configured to recognize a second set of hand gestures using camera data from a higher resolution camera of the wearable device, the second set of hand gestures being larger than the first set of hand gestures.
10. The method of claim 1 , wherein the wearable device comprises a head-wearable device.
11. A computing apparatus comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the apparatus to:
poll a proximity sensor of a wearable device to detect a proximity event, the wearable device comprising a low-power processor and a high-power processor;
in response to detecting the proximity event, perform a gesture detection and recognition routine using a low-power gesture recognition application executed on the low-power processor based on proximity data from the proximity sensor;
determine that the gesture detection and recognition routine executed on the low-power processor fails to recognize a gesture of a user of the wearable device; and
in response to determining that the gesture detection and recognition routine executed on the low-power processor fails to recognize the gesture activate a high-power gesture recognition application executed on the high-power processor.
12. The computing apparatus of claim 11 , wherein the instructions further configure the apparatus to:
generate a level of confidence of the gesture based on a gesture dictionary, using a gesture recognition algorithm operating on the low-power processor; and
identify the gesture in response to the level of confidence exceeding a preset threshold.
13. The computing apparatus of claim 12 , wherein the instructions further configure the apparatus to:
identify an operation corresponding to the gesture; and
request one of the low-power processor or the high-power processor to perform the operation.
14. The computing apparatus of claim 12 , wherein the instructions further configure the apparatus to:
identify an operation of a mixed reality application corresponding to the gesture; and
request the high-power processor to perform the operation of the mixed reality application.
15. The computing apparatus of claim 11 , wherein the instructions further configure the apparatus to:
returning the wearable device to an idle state that polls the proximity sensor at a regular interval after the high-power gesture recognition application recognizes the gesture.
16. The computing apparatus of claim 11 , wherein the instructions further configure the apparatus to:
identify a preset number of gesture recognition tries corresponding to the low-power gesture recognition application, the preset number of gesture recognition tries comprising at least two tries;
determine that the gesture detection and recognition routine executed on the low-power processor fails to recognize the gesture after a number of tries exceeding the preset number of gesture recognition tries,
wherein activating the high-power gesture recognition application executed on the high-power processor is in response to determining that the gesture detection and recognition routine executed on the low-power processor fails to recognize the gesture after the number of tries exceeding the preset number of gesture recognition tries.
17. The computing apparatus of claim 11 , wherein the low-power processor comprises a microcontroller, wherein the high-power processor comprises a system-on-chip (SoC), wherein the proximity sensor comprises a time-of-flight sensor.
18. The computing apparatus of claim 11 , wherein the low-power processor is configured to only operate the proximity sensor and the low-power gesture recognition application, wherein the high-power processor is configured to operate all sensors of the wearable device.
19. The computing apparatus of claim 11 , wherein the low-power gesture recognition application uses a neural network to detect and recognize a hand gesture of a user of the wearable device based on the proximity data, the neural network being configured to recognize a first set of hand gestures, and
wherein the high-power processor comprises a high-power hand-track application configured to recognize a second set of hand gestures using camera data from a higher resolution camera of the wearable device, the second set of hand gestures being larger than the first set of hand gestures.
20. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:
poll a proximity sensor of a wearable device to detect a proximity event, the wearable device comprising a low-power processor and a high-power processor;
in response to detecting the proximity event, perform a gesture detection and recognition routine using a low-power gesture recognition application executed on the low-power processor based on proximity data from the proximity sensor by operating a low power hand tracking application on the low power processor;
determine that the gesture detection and recognition routine executed on the low-power processor fails to recognize a gesture of a user of the wearable device; and
in response to determining that the gesture detection and recognition routine executed on the low-power processor fails to recognize the gesture activate a high-power gesture recognition application executed on the high-power processor.