Granted Patent B2

US 12,211,158 · App. 18/360,945 · Granted Jan 28, 2025

Inventors: Ashwani Arya (Cypress, CA); Alex Feinman (San Mateo, CA); Daniel Harris (San Francisco, CA); Tejas Bahulkar (Sunnyvale, CA); Dunxu Hu (Venice, CA)

Assignee: Snap Inc.

G06T19/006G02B27/017G06F1/325G06F3/017G02B2027/0138

Abstract

Claims (60)

1. A method comprising:

configuring a low-power mode to run on a low-power processor of an AR device using a first set of sensor data from a first set of sensors of the AR device, and a high-power mode to run on a high-power processor of the AR device using a second set of sensor data from a second set of sensors of the AR device;

operating, using the low-power processor, a low-power application in the low-power mode based on the first set of sensor data, wherein the low-power application comprises a low-power navigation application configured to identify a tracking device that generates a beacon signal;

displaying, in a display of the AR device, a visual indicator based on the beacon signal using the low-power processor based on the first set of sensor data; and

detecting that a low-power gesture-based wakeup application operating in the low-power mode failed to detect a hand gesture of a user of the AR device after a preset number of tries; and

in response to detecting that the low-power gesture-based wakeup application failed to detect the hand gesture of the user of the AR device after the preset number of tries, switching from the low-power mode to the high-power mode.

2. The method of claim 1 , further comprising:

detecting a request to operate a high-power application at the AR device;

in response to detecting the request, activating the second set of sensors of the AR device corresponding to the high-power mode;

accessing the second set of sensor data from the second set of sensors with the high-power processor; and

operating, using the high-power processor, the high-power application in the high-power mode based on the second set of sensor data.

3. The method of claim 2 , wherein the low-power application comprises a low-power head up display (HUD) application configured to display virtual content items in the display of the AR device based on the first set of sensor data, the first set of sensor data being independent of a pose of the AR device,

wherein the high-power application comprises an AR application configured to display virtual content items in the display of the AR device based on the second set of sensor data, the second set of sensor data indicating the pose of the AR device based on tracking data from a six-degrees-of-freedom tracking system of the AR device.

4. The method of claim 2 , wherein the low-power application comprises a low-power persistent application that is configured to maintain a wireless connection with a mobile device paired with the AR device and to provide data for processing using the mobile device, and

wherein the high-power application comprises a high-power non-persistent application that is configured to disconnect the AR device from the mobile device and to process data locally on the AR device.

5. The method of claim 1 , wherein the first set of sensors comprises a magnetometer sensor at the AR device.

6. The method of claim 1 , wherein the low-power gesture-based wakeup application uses camera data from a low resolution camera of the AR device, wherein the low-power gesture-based wakeup application is configured to detect the hand gesture of a user of the AR device, wherein the low-power processor is configured to process the camera data and to operate a neural network to recognize a first set of hand gestures, and

wherein a high-power application comprises a high-power gesture-based application configured to recognize a second set of hand gestures using camera data from a higher resolution camera of the AR device using the high-power processor, the second set of hand gestures being larger than the first set of hand gestures.

7. The method of claim 1 , wherein the low-power application comprises a low-power audio application that is configured to: generate audio based on audio data from a mobile device paired with the AR device, to record audio data from a microphone of the AR device, to detect a local voice command based on the audio data, and to offload remote service voice command based on the audio data to the mobile device, and

wherein a high-power application operating on the high-power processor comprises a high-power audio application that is configured to operate a local speech recognition engine with the high-power processor.

8. The method of claim 1 , further comprising:

detecting, with the low-power processor, a pre-defined trigger event corresponding to a high-power application; and

in response to detecting, with the low-power processor, the pre-defined trigger event corresponding to the high-power application, operating the high-power application with the high-power processor.

9. The method of claim 8 , wherein the pre-defined trigger event includes a wake-up word,

wherein the low-power processor includes a MCU processor and the high-power processor includes a SoC processor, and

wherein the AR device is a head-wearable device.

10. A computing apparatus comprising:

one or more processors; and

a memory storing instructions that, when executed by the one or more processors, configure the apparatus to perform operations comprising:

configuring a low-power mode to run on a low-power processor of an AR device using a first set of sensor data from a first set of sensors of the AR device, and a high-power mode to run on a high-power processor of the AR device using a second set of sensor data from a second set of sensors of the AR device;

operating, using the low-power processor, a low-power application in the low-power mode based on the first set of sensor data, wherein the low-power application comprises a low-power navigation application configured to identify a tracking device that generates a beacon signal;

displaying, in a display of the AR device, a visual indicator based on the beacon signal using the low-power processor based on the first set of sensor data; and

detecting that a low-power gesture-based wakeup application operating in the low-power mode failed to detect a hand gesture of a user of the AR device after a preset number of tries; and

in response to detecting that the low-power gesture-based wakeup application failed to detect the hand gesture of the user of the AR device after the preset number of tries, switching from the low-power mode to the high-power mode.

11. The computing apparatus of claim 10 , further comprising:

detecting a request to operate a high-power application at the AR device;

in response to detecting the request, activating the second set of sensors of the AR device corresponding to the high-power mode;

accessing the second set of sensor data from the second set of sensors with the high-power processor; and

operating, using the high-power processor, the high-power application in the high-power mode based on the second set of sensor data.

12. The computing apparatus of claim 11 , wherein the low-power application comprises a low-power head up display (HUD) application configured to display virtual content items in the display of the AR device based on the first set of sensor data, the first set of sensor data being independent of a pose of the AR device,

wherein the high-power application comprises an AR application configured to display virtual content items in a display of the AR device based on the second set of sensor data, the second set of sensor data indicating the pose of the AR device based on tracking data from a six-degrees-of-freedom tracking system of the AR device.

13. The computing apparatus of claim 11 , wherein the low-power gesture-based wakeup application uses camera data from a low resolution camera of the AR device, wherein the low-power gesture-based wakeup application is configured to detect the hand gesture of a user of the AR device, wherein the low-power processor is configured to process the camera data and to operate a neural network to recognize a first set of hand gestures, and

wherein a high-power application comprises a high-power gesture-based application configured to recognize a second set of hand gestures using camera data from a higher resolution camera of the AR device using the high-power processor, the second set of hand gestures being larger than the first set of hand gestures.

14. The computing apparatus of claim 10 , wherein the first set of sensors comprises a magnetometer sensor at the AR device.

15. The computing apparatus of claim 10 , wherein the low-power application comprises a low-power audio application that is configured to: generate audio based on audio data from a mobile device paired with the AR device, to record audio data from a microphone of the AR device, to detect a local voice command based on the audio data, and to offload remote service voice command based on the audio data to the mobile device, and

wherein a high-power application operating on the high-power processor comprises a high-power audio application that is configured to operate a local speech recognition engine with the high-power processor.

16. The computing apparatus of claim 10 , wherein the low-power application comprises a low-power persistent application that is configured to maintain a wireless connection with a mobile device paired with the AR device and to provide data for processing using the mobile device, and

wherein the high-power application comprises a high-power non-persistent application that is configured to disconnect the AR device from the mobile device and to process data locally on the AR device.

17. The computing apparatus of claim 10 , wherein the operations further comprise:

detecting, with the low-power processor, a pre-defined trigger event corresponding to a high-power application; and

in response to detecting, with the low-power processor, the pre-defined trigger event corresponding to a high-power application, operating the high-power application with the high-power processor,

wherein the pre-defined trigger event includes a wake-up word,

wherein the low-power processor includes a MCU processor and the high-power processor includes a SoC processor, and

wherein the AR device is a head-wearable device.

18. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:

configuring a low-power mode to run on a low-power processor of an AR device using a first set of sensor data from a first set of sensors of the AR device, and a high-power mode to run on a high-power processor of the AR device using a second set of sensor data from a second set of sensors of the AR device;

operating, using the low-power processor, a low-power application in the low-power mode based on the first set of sensor data, wherein the low-power application comprises a low-power navigation application configured to identify a tracking device that generates a beacon signal;

displaying, in a display of the AR device, a visual indicator based on the beacon signal using the low-power processor based on the first set of sensor data; and

detecting that a low-power gesture-based wakeup application operating in the low-power mode failed to detect a hand gesture of a user of the AR device after a preset number of tries; and

in response to detecting that the low-power gesture-based wakeup application failed to detect the hand gesture of the user of the AR device after the preset number of tries, switching from the low-power mode to the high-power mode.

Assignments (1)

Continuity (2)

References Cited (30)