Systems, apparatus, and methods for gesture-based augmented reality, extended reality
Systems, apparatus, and methods for a gesture-based augmented reality and/or extended reality (AR/XR) user interface. Conventional image processing scales quadratically based on image resolution. Processing complexity directly corresponds to memory size, power consumption, and heat dissipation. As a result, existing smart glasses solutions have short run-times (<1 hr) and may have battery weight and heat dissipation issues that are uncomfortable for continuous wear. The disclosed solution provides a system and method for low-power image processing via the use of scalable processing. In one specific implementation, gesture detection is divided into multiple stages. Each stage conditionally enables subsequent stages for more complex processing. By scaling processing complexity at each stage, high complexity processing can be performed on an “as-needed” basis.
1. An apparatus for multi-stage hands-free gesture-based user interface processing, comprising:
an eye-tracking camera sensor;
an outward-facing camera sensor;
a power source;
a first processor;
a second processor;
a first non-transitory computer-readable medium comprising a first set of instructions that when executed by the first processor, causes the first processor to:
detect a gesture based on a first image captured by the outward-facing camera sensor;
identify the gesture is a valid gesture associated with one or more gesture-specific tasks;
determine a target based on a second image captured by the eye-tracking camera sensor; and
wake the second processor in response to identification of the valid gesture; and
a second non-transitory computer-readable medium different from the first non-transitory computer-readable medium comprising a second set of instructions that when executed by the second processor, causes the second processor to:
perform the one or more gesture-specific tasks of the valid gesture; and
enter a first sleep state in response to a completion of the one or more gesture-specific tasks.
2. The apparatus of claim 1 , where the first image is captured at a binned resolution and the one or more gesture-specific tasks comprises retrieving a previously captured full resolution image by the outward-facing camera sensor, the previously captured full resolution image captured prior to waking the second processor.
3. The apparatus of claim 1 , where the first image is captured at a binned resolution and the one or more gesture-specific tasks performed by the second processor comprises capturing a full resolution image by the outward-facing camera sensor.
4. The apparatus of claim 1 , further comprising a network interface and a display,
where the one or more gesture-specific tasks comprises downloading media content associated with the target via the network interface rendering the media content via the display.
5. The apparatus of claim 1 , where the second set of instructions further causes the second processor to put the first processor into a second sleep state after the one or more gesture-specific tasks are completed.
6. The apparatus of claim 1 , where the one or more gesture-specific tasks comprises a chained sequence of gestures.
7. A method for multi-stage hands-free gesture-based user interface processing, comprising:
detecting a gesture based on a first image captured by an outward-facing camera sensor of smart glasses, the gesture associated with one or more gesture-specific tasks;
determining a target based on a second image captured by an eye-tracking camera sensor of the smart glasses;
waking a processor of the smart glasses to perform the one or more gesture-specific tasks based on the gesture; and
putting the processor to sleep after the one or more gesture-specific tasks are completed.
8. The method of claim 7 , further comprising:
determining a gaze point captured in at least the second image; and
determining the target based on the gaze point, where the gesture comprises a two-handed gesture.
9. The method of claim 8 , where the gaze point is distinct from the two-handed gesture.
10. The method of claim 8 , where:
the two-handed gesture comprises a frame formed by hands of a user, and
determining the target comprises determining the gaze point is not on or within the frame created by the two-handed gesture.
11. The method of claim 8 , where the two-handed gesture comprises a shorthand finger positioning.
12. The method of claim 7 , where the target is determined based on a gaze point captured in at least the second image and where the gesture comprises a one-handed gesture.
13. The method of claim 12 , where the one-handed gesture comprises a shorthand finger positioning.
14. The method of claim 7 , further comprising determining an indistinct target based on determining there is an indistinct gaze point or a gaze point fixated on one or both hands of a user.
15. An apparatus for multi-stage hands-free gesture-based user interface processing, comprising:
an eye-tracking camera sensor comprising a first neural network trained to determine a target based on a first image captured by the eye-tracking camera sensor;
an outward-facing camera sensor comprising a second neural network trained to detect a gesture based on a second image captured by the outward-facing camera sensor;
a power source;
a processor; and
a non-transitory computer-readable medium comprising a set of instructions that when executed by the processor, causes the processor to:
perform one or more gesture-specific tasks based on the target determined by the eye-tracking camera sensor and the gesture determined by the outward-facing camera sensor; and
enter a first sleep state after the one or more gesture-specific tasks are completed.
16. The apparatus of claim 15 , further comprising a network interface and a display,
where the one or more gesture-specific tasks comprises downloading media content associated with the target via the network interface and rendering the media content via the display.
17. The apparatus of claim 15 , where the set of instructions further causes the processor to put the outward-facing camera sensor into a second sleep state after the one or more gesture-specific tasks are completed.
18. The apparatus of claim 15 , where:
the second neural network of the outward-facing camera sensor is trained to detect an initial gesture associated with a chained sequence of gestures, and
the one or more gesture-specific tasks performed by the processor comprises detecting the chained sequence of gestures in response to detection of the initial gesture by the outward-facing camera sensor, each gesture of the chained sequence of gestures comprising separate actions and associated with separate gesture specific tasks based on previously performed gestures of the chained sequence of gestures.
19. The apparatus of claim 15 , where the target is determined based on a gaze point, the gesture comprises a two-handed gesture, and the gaze point is distinct from the two-handed gesture.
20. The apparatus of claim 15 , where the target is determined based on a gaze point, the gesture comprises a one-handed gesture, and the gaze point is indistinct from the one-handed gesture.