Granted Patent B2

US 12,645,566 · App. 17/203,681 · Granted Jun 2, 2026

Inventors: Tyler L. Casella (San Jose, CA); Norman N. Wang (Saratoga, CA); Benjamin Breckin Loggins (Santa Clara, CA); Daniel M. Delwood (San Jose, CA)

G06F11/362G02B27/017G06F11/3698G06T15/06G06T19/006

Abstract

Claims (56)

1 . A system comprising:

a display;

one or more motion sensors attached to the display; and

a processing apparatus configured to:

associate a breakpoint with a set of views that corresponds to a region in a simulated reality environment presented using the display;

access motion data captured using the one or more motion sensors;

determine, based at least on the motion data, a view within the simulated reality environment, wherein the view is stored as a viewing vector from a position in the simulated reality environment;

based on the stored viewing vector, detect whether the view is a member of the set of views associated with the breakpoint;

responsive to detecting that the view is a member of the set of views associated with the breakpoint, trigger the breakpoint and perform a debug action associated with the breakpoint;

while performing the debug action, continue executing a simulation process of the simulated reality environment, wherein a state of at least one virtual object in the simulated reality environment continues to be viewed with the display and continues to respond to input received using the one or more motion sensors; and

wherein the processing apparatus is configured to detect that the view is associated with the breakpoint by determining an angle between the viewing vector from the position in the simulated reality environment and a central viewing vector.

2 . The system of claim 1 , wherein the set of views corresponds to a range of angles with respect to the central viewing vector, and the processing apparatus is configured to detect that the view is associated with the breakpoint by:

determining whether the position in the simulated reality environment is within the region in the simulated reality environment;

and

determining whether the angle is within the range of angles.

3 . The system of claim 1 , wherein the view corresponds to a ray within the simulated reality environment, the set of views corresponds to a set of rays that pass through the region in the simulated reality environment, and the processing apparatus is configured to detect that the view is associated with the breakpoint by:

determining whether the ray of the view passes through the region of the set of views.

4 . The system of claim 1 , wherein the view corresponds to a focal point within the simulated reality environment, and the processing apparatus is configured to detect that the view is associated with the breakpoint by:

determining whether the focal point of the view is within the region corresponding to the set of views.

5 . The system of claim 1 , wherein the breakpoint has multiple trigger conditions and the processing apparatus is configured to:

trigger the breakpoint based at least on a line of code in a process associated with a virtual object in the simulated reality environment being reached and the view concurrently being associated with the breakpoint.

6 . The system of claim 1 , wherein the breakpoint has multiple trigger conditions and the processing apparatus is configured to:

trigger the breakpoint based at least on a condition on a state of a virtual object in the simulated reality environment being met and the view concurrently being associated with the breakpoint.

7 . The system of claim 1 , wherein the breakpoint has multiple trigger conditions and the processing apparatus is configured to:

trigger the breakpoint based at least on detecting an action by a user of the display and the view concurrently being associated with the breakpoint.

8 . The system of claim 1 , wherein the processing apparatus is configured to:

responsive to the breakpoint being triggered, identify one or more objects in the simulated reality environment to be debugged by ray casting based at least on the view; and perform the debug action on one or more processes associated with the one or more identified objects.

9 . The system of claim 1 , wherein the debug action includes pausing execution of a process associated with a virtual object in the simulated reality environment.

10 . The system of claim 9 , wherein the processing apparatus is configured to:

generate a visual representation of the object associated with the paused process; and

continue to present, using the display, the visual representation within the simulated reality environment while the process is paused.

11 . The system of claim 1 , wherein the debug action includes presenting, using the display, a debug control interface within the simulated reality environment that enables a user of the display to control execution of a process associated with a virtual object in the simulated reality environment.

12 . The system of claim 1 , wherein the debug action includes presenting, using the display, within the simulated reality environment, code of a process associated with a virtual object in the simulated reality environment.

13 . The system of claim 1 , wherein the display is a head-mounted display.

14 . A method comprising:

associating a breakpoint with a set of views that corresponds to a region in a simulated reality environment presented using a display;

accessing motion data captured using one or more motion sensors;

determining, based at least on the motion data, a view within the simulated reality environment, wherein the view is stored as a viewing vector from a position in the simulated reality environment;

based on the stored viewing vector, detecting whether the view is a member of the set of views associated with the breakpoint;

responsive to detecting that the view is a member of the set of views associated with the breakpoint, triggering the breakpoint;

responsive to the breakpoint being triggered, performing a debug action associated with the breakpoint;

while performing the debug action, continuing to execute a simulation process of the simulated reality environment, wherein a state of at least one virtual object in the simulated reality environment continues to evolve and be viewed with the display and continues to respond to input received using the one or more motion sensors; and

wherein detecting that the view is associated with the breakpoint comprises: determining an angle between the viewing vector from the position in the simulated reality environment and a central viewing vector.

15 . The method of claim 14 , wherein the set of views corresponds to a range of angles with respect to the central viewing vector, and detecting that the view is associated with the breakpoint comprises:

determining whether the position in the simulated reality environment is within the region in the simulated reality environment;

determining whether the angle is within the range of angles.

16 . The method of claim 14 , wherein the view corresponds to a ray within the simulated reality environment, the set of views corresponds to a set of rays that pass through the region in the simulated reality environment, and detecting that the view is associated with the breakpoint comprises:

determining whether the ray of the view passes through the region of the set of views.

17 . The method of claim 14 , wherein the view corresponds to a focal point within the simulated reality environment, and detecting that the view is associated with the breakpoint comprises:

determining whether the focal point of the view is within the region corresponding to the set of views.

18 . The method of claim 14 , comprising:

triggering the breakpoint based at least on a line of code in a process associated with a virtual object in the simulated reality environment being reached and the view concurrently being associated with the breakpoint.

19 . The method of claim 14 , comprising:

responsive to the breakpoint being triggered, identifying one or more objects in the simulated reality environment to be debugged by ray casting based at least on the view; and

performing the debug action on one or more processes associated with the one or more identified objects.

20 . The method of claim 14 , wherein the debug action includes pausing execution of a process associated with a virtual object in the simulated reality environment.

Continuity (4)

References Cited (39)