Systems, apparatus, and methods for gesture-based augmented reality, extended reality
Systems, apparatus, and methods for a gesture-based augmented reality and/or extended reality (AR/XR) user interface. Conventional image processing scales quadratically based on image resolution. Processing complexity directly corresponds to memory size, power consumption, and heat dissipation. As a result, existing smart glasses solutions have short run-times (<1 hr) and may have battery weight and heat dissipation issues that are uncomfortable for continuous wear. The disclosed solution provides a system and method for low-power image processing via the use of scalable processing. In one specific implementation, gesture detection is divided into multiple stages. Each stage conditionally enables subsequent stages for more complex processing. By scaling processing complexity at each stage, high complexity processing can be performed on an “as-needed” basis.
1. A gesture-driven scalable processing apparatus, comprising:
a scalable processing subsystem comprising a machine learning processor, and a power source configured to provide power in a standby state, a capture state, and a context state; and
a non-transitory computer-readable medium comprising a first set of instructions that when executed by the scalable processing subsystem, causes the scalable processing subsystem to:
train the machine learning processor to recognize a gaze fixation event and a hand motion event;
obtain eye-tracking images at a first frame rate in the standby state;
cause the power source to change to the capture state when the gaze fixation event is detected within the eye-tracking images by the machine learning processor;
obtain front-facing images at a second frame rate in the capture state; and
cause the power source to change to the context state when the hand motion event is detected within the front-facing images by the machine learning processor.
2. The gesture-driven scalable processing apparatus of claim 1 , where the machine learning processor is trained during an offline set-up sequence that prompts a user to perform a plurality of user interactions.
3. The gesture-driven scalable processing apparatus of claim 1 , where the machine learning processor is trained to recognize at least one of a physical speed and a range of motion of the hand motion event.
4. The gesture-driven scalable processing apparatus of claim 1 , where the machine learning processor is trained to recognize the gaze fixation event based on at least one of a threshold duration or a threshold amplitude.
5. The gesture-driven scalable processing apparatus of claim 1 , where the first set of instructions, when executed by the scalable processing subsystem, further causes the scalable processing subsystem to determine a gesture based on at least one of a gaze point, a region of interest, or a hand movement in the context state.
6. The gesture-driven scalable processing apparatus of claim 5 , where the scalable processing subsystem is trained to recognize a set of gestures that can be combined into a larger set of gestures.
7. The gesture-driven scalable processing apparatus of claim 5 , where the first set of instructions, when executed by the scalable processing subsystem, further causes the scalable processing subsystem to cause the power source to return to the standby state when the hand motion event is not detected within the front-facing images by the machine learning processor.
8. A method for gesture-driven scalable processing, comprising:
training a machine learning processor to recognize a first user interaction at a first exposure setting and a first frame rate and a second user interaction at a second exposure setting and a second frame rate;
capturing a first set of images in a standby power state at the first exposure setting and the first frame rate;
changing a scalable processing subsystem to a capture power state when the first user interaction is detected by the machine learning processor;
capturing a second set of images in the capture power state at the second exposure setting and the second frame rate; and
changing the scalable processing subsystem to a context power state when the second user interaction is detected by the machine learning processor.
9. The method of claim 8 , where the first set of images are captured by an inward-facing camera sensor and the first user interaction comprises a gaze fixation event.
10. The method of claim 9 , where the machine learning processor is trained to detect the gaze fixation event based on at least one of a threshold duration or a threshold amplitude.
11. The method of claim 8 , where the first set of images are captured by an outward-facing camera sensor and the first user interaction comprises a hand motion event.
12. The method of claim 11 , where the machine learning processor is trained to detect the hand motion event based on at least one of a physical speed and a range of motion of a user's hand.
13. The method of claim 8 , further comprising determining a gesture based on at least one of a gaze point, a region of interest, or a hand movement in the context power state.
14. The method of claim 8 , further comprising capturing a sequential chain of additional gestures during the context power state.
15. A scalable processing subsystem configured to:
train a machine learning processor to recognize a set of user-specific user interactions captured at a plurality of resolutions and image qualities, the set of user-specific user interactions associated with a corresponding set of power states;
cause the scalable processing subsystem to change to a second power state when a first user-specific user interaction is recognized during a first power state;
cause the scalable processing subsystem to change to a third power state when a second user-specific user interaction is recognized during the second power state; and
cause the scalable processing subsystem to change to a fourth power state when a third user-specific user interaction is recognized during the third power state.
16. The scalable processing subsystem of claim 15 , further configured to:
cause the scalable processing subsystem to return to the first power state when the second user-specific user interaction is not recognized during the second power state; and
cause the scalable processing subsystem to return to the second power state when the third user-specific user interaction is not recognized during the third power state.
17. The scalable processing subsystem of claim 15 , further configured to:
cause the scalable processing subsystem to return to the first power state when the second user-specific user interaction is not recognized during the second power state; and
cause the scalable processing subsystem to return to the first power state when the third user-specific user interaction is not recognized during the third power state.
18. The scalable processing subsystem of claim 15 , where the machine learning processor is trained during an offline set-up sequence that prompts a user to perform a plurality of user interactions.
19. The scalable processing subsystem of claim 18 , where the plurality of user interactions is a subset of, but can be combined into, the set of user-specific user interactions associated with the corresponding set of power states.
20. The scalable processing subsystem of claim 15 , where the machine learning processor is trained to recognize the set of user-specific user interactions at a resolution, frame rate, exposure setting or image quality associated with the corresponding set of power states.