Granted Patent B2

US 12,593,023 · App. 18/530,642 · Granted Mar 31, 2026

Inventors: Mohamed Al Sharnouby (Dublin, CA); Arun Kannan (Mountain View, CA); Venu M Duggineni (San Jose, CA); Kaushik Raghunath (Pleasanton, CA); Saul H Weiss (Parker, CO); Luke Yoder (Poulsbo, WA); James C McIlree (Santa Clara, CA); Sankaravadivoo Subramanian (Saratoga, CA); Mukta S Gore (Sunnyvale, CA); Russell L Jones (Westminster, CO)

Assignee: Apple Inc.

H04N13/383G06F3/013H04N13/158

Abstract

Claims (85)

1 . A method of operating an electronic device comprising:

with at least one image sensor, acquiring a video feed;

with a scheduler running in an application layer of the electronic device, obtaining a description of tasks to be performed by a plurality of applications running in the application layer;

determining whether to operate the electronic device in a first video passthrough mode or a second video passthrough mode based on the description of tasks;

while operating in the first video passthrough mode, displaying a passthrough video feed by modifying the video feed using information from the applications; and

while operating in the second video passthrough mode, displaying a passthrough video feed without modifying the video feed using the information from the applications.

2 . The method of claim 1 , wherein the applications comprise applications configured to generate extended reality content, the method further comprising:

in response to determining that the applications configured to generate the extended reality content is failing, triggering a configurable corrective action to mitigate the failure.

3 . The method of claim 1 , further comprising:

with a kernel driver running in a kernel layer of the electronic device, receiving keep alive messages from the scheduler; and

in response to the kernel driver not receiving any keep alive messages from the scheduler for a period of time exceeding a threshold, switching the electronic device between the first video passthrough mode and the second video passthrough mode.

4 . The method of claim 3 , further comprising:

with a central processing unit (CPU) in the electronic device, receiving keep alive messages from the kernel driver; and

in response to the CPU not receiving any keep alive messages from the kernel driver for a period of time exceeding a threshold or in response to the CPU receiving a kernel panic signal from the kernel driver, switching the electronic device between the first video passthrough mode and the second video passthrough mode.

5 . The method of claim 3 , further comprising:

with the kernel driver, receiving keep alive messages from a central processing unit (CPU) in the electronic device; and

in response to the kernel driver not receiving any keep alive messages from the CPU for a period of time exceeding a threshold or in response to the kernel driver receiving a panic signal from the CPU, rebooting the electronic device.

6 . The method of claim 4 , further comprising:

with the CPU, receiving keep alive messages from at least one input-output processor in the electronic device; and

in response to the CPU not receiving any keep alive messages from the at least one input-output processor for a period of time exceeding a threshold or in response to the CPU receiving a panic signal from the at least one input-output processor, switching the electronic device between the first video passthrough mode and the second video passthrough mode.

7 . The method of claim 4 , further comprising:

with a first data transport controller, transmitting video frames to a second data transport controller; and

in response to the second data transport controller detecting an error in the transmitted video frames, asserting an error signal to alert the CPU.

8 . The method of claim 1 , further comprising:

determining whether to operate the electronic device in the first video passthrough mode, the second video passthrough mode, or a third video passthrough mode based on the description of tasks;

while operating in the third video passthrough mode, displaying a passthrough video feed by modifying the video feed using information from some of the applications while bypassing information from at least one failing application in the plurality of applications;

with a watchdog timer, monitoring whether the electronic device has been stuck in the second video passthrough mode or the third video passthrough mode for a period of time exceeding a threshold; and

in response to the watchdog timer expiring, performing a system reboot.

9 . A method of operating an electronic device comprising:

with at least one image sensor, acquiring a video feed;

with a scheduler running in an application layer of the electronic device, obtaining a description of tasks to be performed by a plurality of applications running in the application layer by receiving compute graphs from the plurality of applications and analyzing the compute graphs to identify a condition of the applications; and

determining whether to operate the electronic device in a first video passthrough mode or a second video passthrough mode based on the description of tasks.

10 . The method of claim 9 , wherein the applications comprise a gaze tracking application configured to execute a gaze tracking algorithm that tracks a user's gaze, and wherein the scheduler is configured to detect a failure of the gaze tracking algorithm by monitoring an output of a node in one of the compute graphs associated with the gaze tracking algorithm.

11 . The method of claim 10 , further comprising:

in response to detecting a failure of the gaze tracking algorithm, disabling dynamic foveation while operating in the first or second video passthrough mode.

12 . The method of claim 9 , wherein the applications comprise a head pose tracking application configured to execute a head pose tracking algorithm that tracks a user's head pose, and wherein the scheduler is configured to detect a failure of the head pose tracking algorithm by monitoring an output of a node in one of the compute graphs associated with the head pose tracking algorithm.

13 . The method of claim 9 , wherein the applications comprise a virtual content compositor configured to execute a virtual content compositor algorithm that generates composited virtual frames based on user tracking data, and wherein the scheduler is configured to detect a failure of the virtual content compositor algorithm by monitoring an output of a node in one of the compute graphs associated with the virtual content compositor algorithm.

14 . The method of claim 9 , wherein the applications comprise a graphics rendering application configured to execute a graphics rendering algorithm that generates virtual content, and wherein the scheduler is configured to detect a failure of the graphics rendering algorithm by monitoring an output of a node in one of the compute graphs associated with the graphics rendering algorithm.

15 . A method of operating an electronic device comprising:

with at least one image sensor, acquiring a video feed;

with a scheduler running in an application layer of the electronic device, obtaining a description of tasks to be performed by a plurality of applications running in the application layer;

determining whether to operate the electronic device in a first video passthrough mode or a second video passthrough mode based on the description of tasks;

in response to determining that a first application in the plurality of applications is failing, triggering a first configurable corrective action; and

in response to determining that a second application, different than the first application, in the plurality of applications is failing, triggering a second configurable corrective action different than the first configurable corrective action.

16 . A method of operating an electronic device comprising:

with at least one image sensor, acquiring a video feed;

monitoring a condition of a first subsystem using a second subsystem;

determining whether to operate the electronic device in a first video passthrough mode or a second video passthrough mode based on the condition of the first subsystem as monitored by the second subsystem;

monitoring a condition of the second subsystem using a third subsystem; and

determining whether to operate the electronic device in the first video passthrough mode or the second video passthrough mode based on the condition of the second subsystem as monitored by the third subsystem.

17 . The method of claim 16 , further comprising:

while operating in the first video passthrough mode, displaying a passthrough video feed by using processing circuitry to perform a processing function on the video feed from the at least one image sensor; and

while operating in the second video passthrough mode, displaying a passthrough video feed based on the video feed from the at least one image sensor without performing the processing function.

18 . The method of claim 16 , further comprising:

monitoring a condition of the third subsystem using a fourth subsystem; and

determining whether to operate the electronic device in the first video passthrough mode or the second video passthrough mode based on the condition of the third subsystem as monitored by the fourth subsystem.

19 . The method of claim 16 , wherein:

the second subsystem comprises a scheduling and resource management subsystem configured to run in an application layer of the electronic device, to receive compute graphs from a plurality of applications running in the application layer, and to analyze the compute graphs to monitor a condition of the applications; and

the first subsystem comprises one of the plurality of applications selected from the group consisting of: a graphics renderer, a virtual content compositor, a gaze tracker, and a hands tracker.

20 . The method of claim 16 , wherein:

the second subsystem comprises a scheduling and resource management subsystem configured to run in an application layer of the electronic device, to receive compute graphs from a plurality of applications running in the application layer, and to analyze the compute graphs to monitor a condition of the applications; and

the third subsystem comprises a kernel driver configured to run in a kernel layer of the electronic device and to receive keep alive messages from the scheduling the resource management subsystem to monitor the condition of the second subsystem.

21 . The method of claim 16 , wherein:

the second subsystem comprises a kernel driver configured to run in a kernel layer of the electronic device; and

the third subsystem comprises a central processing unit (CPU) of the electronic device configured to receive keep alive messages or a kernel panic signal from the kernel driver to monitor the condition of the second subsystem.

22 . The method of claim 16 , wherein the third subsystem comprises a central processing unit (CPU) of the electronic device, the method further comprising:

monitoring a condition of the third subsystem using a fourth subsystem, wherein the fourth subsystem comprises a kernel driver configured to run in a kernel layer of the electronic device and configured to receive keep alive messages or a panic signal from the CPU to monitor the condition of the third subsystem; and

rebooting at least the CPU based on the condition of the third subsystem as monitored by the fourth subsystem.

23 . The method of claim 16 , wherein:

the second subsystem comprises an input-output processor of the electronic device; and

the third subsystem comprises a central processing unit (CPU) of the electronic device configured to receive keep alive messages or a panic signal from the input-output processor to monitor the condition of the second subsystem.

24 . The method of claim 23 , further comprising:

with a first data transport controller, transmitting video frames to a second data transport controller; and

in response to the second data transport controller detecting an error in the transmitted video frames, asserting an error signal to alert the CPU.

25 . The method of claim 16 , further comprising:

in response to a watchdog timer expiring, performing a system reboot.

26 . The method of claim 16 , wherein monitoring a condition of the first subsystem comprises determining whether a number of repeating frames output from the first subsystem exceeds a threshold number of repeating frames, and wherein the threshold number of repeating frames is adjustable based on historical data.

27 . An electronic device comprising:

at least one image sensor configured to acquire a video feed;

a first subsystem;

a second subsystem configured to monitor a condition of the first subsystem;

a third subsystem configured to monitor a condition of the second subsystem; and

one or more displays configured to operate in a first video passthrough mode or a second video passthrough mode based on the condition of the first subsystem as monitored by the second subsystem and based on the condition of the second subsystem as monitored by the third subsystem, wherein

in the first video passthrough mode, the one or more displays is configured to output a passthrough video feed by using processing circuitry to perform a processing function on the video feed from the at least one image sensor, and

in the second video passthrough mode, the one or more displays is configured to output a passthrough video feed based on the video feed from the at least one image sensor without performing the processing function.

Assignments (1)

Continuity (2)

References Cited (21)