Systems, apparatus, and methods for gesture-based augmented reality, extended reality
Systems, apparatus, and methods for a gesture-based augmented reality and/or extended reality (AR/XR) user interface. Conventional image processing scales quadratically based on image resolution. Processing complexity directly corresponds to memory size, power consumption, and heat dissipation. As a result, existing smart glasses solutions have short run-times (<1 hr) and may have battery weight and heat dissipation issues that are uncomfortable for continuous wear. The disclosed solution provides a system and method for low-power image processing via the use of scalable processing. In one specific implementation, gesture detection is divided into multiple stages. Each stage conditionally enables subsequent stages for more complex processing. By scaling processing complexity at each stage, high complexity processing can be performed on an “as-needed” basis.
1. A gesture-driven scalable processing apparatus, comprising:
a physical frame configured to hold the gesture-driven scalable processing apparatus within a fixed location relative to a user head;
a scalable processing subsystem comprising at least a machine learning processor, and a power source configured to provide power in a plurality of power states;
a camera sensor logic in communication with the machine learning processor; and
a non-transitory computer-readable medium comprising a first set of instructions that when executed by the camera sensor logic, causes the camera sensor logic to:
train the machine learning processor to recognize a plurality of user-specific user interactions during an offline set-up sequence, where the offline set-up sequence prompts a user to perform a plurality of user interactions;
capture a plurality of images at a first resolution and a first frame rate in a first power state; and
cause the power source to change to a second power state when a first user interaction of the plurality of user-specific user interactions is detected within at least a first image by the machine learning processor.
2. The gesture-driven scalable processing apparatus of claim 1 , where the plurality of user interactions comprises user-specific mannerisms.
3. The gesture-driven scalable processing apparatus of claim 2 , where the user-specific mannerisms are treated as shorthand finger positionings.
4. The gesture-driven scalable processing apparatus of claim 1 , where the plurality of user interactions comprises at least one hand motion at a plurality of speeds.
5. The gesture-driven scalable processing apparatus of claim 4 , where the first resolution and the first frame rate are determined based on the plurality of speeds.
6. A method for gesture-driven scalable processing in smart glasses, comprising:
training a machine learning processor to recognize a plurality of user-specific user interactions at a first exposure setting and a first frame rate, based on at least one of a physical speed and a range of motion of a user hand;
capturing a plurality of images with a camera sensor in a first power state at the first exposure setting and the first frame rate; and
changing the camera sensor to a second power state when a first user interaction of the plurality of user-specific user interactions is detected by the machine learning processor.
7. The method of claim 6 , where the plurality of user-specific user interactions include comprise at least one user-specific idiosyncrasy or cultural mannerism.
8. The method of claim 6 , where the first frame rate is based on the physical speed or the range of motion.
9. The method of claim 6 , where the machine learning processor is trained to recognize the plurality of user-specific user interactions based on at least one of a shape, a complexion, a size, and an orientation of the user hand.
10. The method of claim 9 , where the first exposure setting is based on the shape, the complexion, the size, or the orientation.
11. The method of claim 6 , where the plurality of user-specific user interactions comprises a one-handed gesture.
12. The method of claim 6 , where the plurality of user-specific user interactions comprises a two-handed gesture.
13. The method of claim 6 , where the plurality of user-specific user interactions comprises a finger positioning.
14. A gesture-driven scalable processing apparatus, comprising:
a camera sensor logic configured to capture a plurality of images at a first frame rate;
a machine learning processor; and
a non-transitory computer-readable medium comprising a first set of instructions that when executed by the machine learning processor, causes the machine learning processor to:
learn to recognize a plurality of user-specific user interactions at the first frame rate during a set-up sequence, where a set of user-specific user interactions comprise at least one two-handed position, one-handed position, or a gaze fixation that can be combined into a larger set of gestures;
obtain the plurality of images at the first frame rate during an operational mode; and
cause a power source to change power state when a first user interaction of the plurality of user-specific user interactions is detected within at least a first image by the machine learning processor.
15. The gesture-driven scalable processing apparatus of claim 14 , where the camera sensor logic comprises an inward-facing camera sensor and an outward-facing camera sensor.
16. The gesture-driven scalable processing apparatus of claim 14 , where further comprising a physical frame configured to hold the gesture-driven scalable processing apparatus within a fixed location relative to a user head.
17. The gesture-driven scalable processing apparatus of claim 14 , where the power source is further configured to provide different power levels in a standby state, a capture state, a context state, and a gesture state.