Granted Patent B2

US 12,293,013 · App. 17/726,390 · Granted May 6, 2025

Inventors: Alysha Naples (London, GB); Jonathan Lawrence Mann (Seattle, WA); Paul Armistead Hoover (Bothell, WA)

Assignee: Magic Leap, Inc.

G06F3/011G06F3/012G06F3/013G06F3/04815G06V20/20G06V40/193

Abstract

Claims (97)

1. A method for providing an indication regarding presence of a virtual object in an augmented reality environment around a user, the method comprising: under control of an augmented reality (AR) system comprising computer hardware, the AR system configured to permit user interaction with virtual objects in a field of regard (FOR) of the user, the FOR comprising a portion of the environment around the user that is capable of being perceived by the user via the AR system:

determining a group of virtual objects in the FOR of the user;

determining a pose of the user;

determining a field of view (FOV) of the user based at least partly on the pose of the user, the FOV comprising a portion of the FOR that is capable of being perceived at a given time by the user via the AR system;

identifying a subgroup of the group of virtual objects that are located inside of the FOR of the user but outside of the FOV of the user, the subgroup including at least one interactable virtual object;

for one or more of the virtual objects in the subgroup of virtual objects:

determining a location of the virtual object relative to the FOV of the user;

determining, based at least in part on the location, a placement of a visual aura associated with the virtual object relative to the FOV of the user; and

displaying the visual aura such that at least a portion of the visual aura is perceivable by the user to be on an edge of the FOV of the user,

wherein the visual aura represents contextual information associated with the at least one interactable virtual object in the subgroup and further comprising:

accessing the contextual information associated with the at least one interactable virtual object, and

dynamically changing the visual aura when the contextual information changes, wherein the contextual information comprises received messages, the change in the contextual information comprises an increased number of received messages and the dynamically changing the visual aura comprises the aura becoming thicker.

2. The method of claim 1 , wherein the pose of the user comprises an eye pose of the user.

3. The method of claim 1 , wherein the visual aura comprises a shape having a color.

4. The method of claim 3 , wherein the shape comprises a rounded square.

5. The method of claim 3 , wherein the color of the visual aura indicates a type of the virtual object.

6. The method of claim 1 , wherein the placement of the aura comprises one or more of the following: brightness, position, or size.

7. The method of claim 6 , wherein the aura comprises brightness and the brightness is based at least in part on proximity of the virtual object relative to the edge of the FOV.

8. The method of claim 1 , wherein the FOV of the user is determined based at least partly on an area of an AR display in the AR system.

9. The method of claim 1 , wherein the virtual objects comprise one or more of the following: an operating system virtual object or an application virtual object.

10. The method of claim 1 , further comprising: updating the subgroup of the group of virtual objects based at least partly on a change of the pose of the user; and updating the first or second representation of the visual aura based on the updated subgroup of the group of virtual objects.

11. The method of claim 1 , further comprising: determining that a virtual object in the subgroup of virtual objects has moved inside of the FOV of the user; and ceasing to display the visual aura associated with that virtual object.

12. The method of claim 1 , further comprising: determining that a virtual object in the FOV of the user has moved outside of the FOV of the user; and displaying a visual aura associated with that virtual object.

13. The method of claim 1 , further comprising identifying a physical group of physical objects that are located inside of the FOR and inside the FOV of the user, the physical group including at least one interactable physical object.

14. The method of claim 1 , further comprising identifying a virtual group of virtual objects that are located inside of the FOR and inside the FOV of the user, the virtual group including at least one interactable FOV object and further comprising interacting with the at least one interactable FOV object.

15. The method of claim 14 , wherein the interactable FOV object comprises a virtual graph of a change in stock prices over time, and further comprising interacting with the virtual graph including buying stock.

16. The method of claim 15 , wherein said interacting with the virtual graph comprises hand gestures of the user.

17. The method of claim 15 , wherein said interacting with the virtual graph comprises the user tapping a foot.

18. The method of claim 1 , further comprising identifying at least one physical object in said FOR; and, adding a virtual element to the physical object.

19. The method of claim 1 , wherein the displaying the visual aura comprises displaying the visual aura on a virtual user interface (UI) and further comprising interacting with the virtual UI using a totem, to generate more virtual content on the virtual UI.

20. The method of claim 1 , further comprising, for at least one virtual object in the subgroup of virtual objects, informing the user of the at least one virtual object using a tactile effect.

21. The method of claim 1 , wherein the visual aura represents a two-dimensional projection of the virtual object.

22. The method of claim 1 , further comprising identifying a physical object located inside of the FOR of the user but outside of the FOV of the user, determining a location of the physical object relative to the FOV of the user and displaying a visual aura associated with the virtual object relative to the FOV of the user such that a portion of the visual aura is perceivable by the user to be on an edge of the FOV of the user, wherein the visual aura represents a 2 dimensional projection of the physical object.

23. The method of claim 1 , further comprising providing a visual aura associated with an object that is located outside the FOR of the user and displaying the visual aura such that at least a portion of the visual aura is perceivable by the user to be on an edge of the FOV of the user.

24. A method for providing an indication regarding presence of a virtual object in an augmented reality environment around a user, the method comprising: under control of an augmented reality (AR) system comprising computer hardware, the AR system configured to permit user interaction with virtual objects in a field of regard (FOR) of the user, the FOR comprising a portion of the environment around the user that is capable of being perceived by the user via the AR system:

determining a group of virtual objects in the FOR of the user;

determining a pose of the user;

determining a field of view (FOV) of the user based at least partly on the pose of the user, the FOV comprising a portion of the FOR that is capable of being perceived at a given time by the user via the AR system;

identifying a subgroup of the group of virtual objects that are located inside of the FOR of the user but outside of the FOV of the user, the subgroup including at least one interactable virtual object;

for one or more of the virtual objects in the subgroup of virtual objects:

determining a location of the virtual object relative to the FOV of the user;

determining, based at least in part on the location, a placement of a visual aura associated with the virtual object relative to the FOV of the user; and

displaying the visual aura such that at least a portion of the visual aura is perceivable by the user to be on an edge of the FOV of the user,

wherein the visual aura represents contextual information associated with the at least one interactable virtual object in the subgroup and further comprising;

accessing the contextual information associated with the at least one interactable virtual object, and

dynamically changing the visual aura when the contextual information changes,

wherein the contextual information comprises location of the object and wherein the change in the contextual information comprises change in proximity to the FOV and the dynamically changing the visual aura comprises the aura thinning, and further comprising;

identifying a physical group of physical objects that are located inside of the FOR and inside the FOV of the user, the physical group including at least one interactable physical object.

25. A method for providing an indication regarding presence of a virtual object in an augmented reality environment around a user, the method comprising: under control of an augmented reality (AR) system comprising computer hardware, the AR system configured to permit user interaction with virtual objects in a field of regard (FOR) of the user, the FOR comprising a portion of the environment around the user that is capable of being perceived by the user via the AR system:

determining a group of virtual objects in the FOR of the user;

determining a pose of the user;

determining a field of view (FOV) of the user based at least partly on the pose of the user, the FOV comprising a portion of the FOR that is capable of being perceived at a given time by the user via the AR system;

identifying a subgroup of the group of virtual objects that are located inside of the FOR of the user but outside of the FOV of the user, the subgroup including at least one interactable virtual object;

for one or more of the virtual objects in the subgroup of virtual objects:

determining a location of the virtual object relative to the FOV of the user;

determining, based at least in part on the location, a placement of a visual aura associated with the virtual object relative to the FOV of the user; and

displaying the visual aura such that at least a portion of the visual aura is perceivable by the user to be on an edge of the FOV of the user,

wherein the visual aura represents contextual information associated with the at least one interactable virtual object in the subgroup and further comprising:

accessing the contextual information associated with the at least one interactable virtual object, and

dynamically changing the visual aura when the contextual information changes,

wherein the method comprises displaying a first AR display for a first eye of the user and displaying a second, different AR display for a second eye of the user, and wherein displaying the visual aura on the edge of the FOV of the user comprises displaying a first representation of the visual aura by the first AR display, and further comprising

identifying at least one physical object in said FOR; and, adding a virtual element to the physical object,

wherein one of

a) the contextual information comprises location of the object and wherein the change in the contextual information comprises change in proximity to the FOV and the dynamically changing the visual aura comprises the aura thinning, and

b) the contextual information comprises received messages, the change in the contextual information comprises an increased number of received messages and the dynamically changing the visual aura comprises the aura becoming thicker.

26. The method of claim 25 , further comprising displaying a second representation of the visual aura by the second AR display, the second representation different from the first representation.

27. The method of claim 26 , wherein the first representation of the visual aura and the second representation of the visual aura are rendered separately to each eye to match peripheral vision or to reduce or avoid depth perception of the visual aura.

28. The method of claim 25 , further comprising identifying a virtual group of virtual objects that are located inside of the FOR and inside the FOV of the user, the virtual group including at least one interactable FOV object, wherein the interactable FOV object comprises a virtual graph of a change in stock prices over time, and further comprising the user interacting with the virtual graph including buying stock by pointing of the user's arms.

29. The method of claim 25 , wherein the physical object is a television and adding a virtual element comprises adding a virtual menu associated with the television, the virtual menu enabling the user to turn on or change channels of the television using the wearable system.

30. A method for providing an indication of an object in an environment of a user, the method comprising: under control of an augmented reality (AR) system comprising an imaging system, the AR system configured to permit user interaction with virtual objects in a field of regard (FOR) of the user, the FOR comprising a portion of the environment around the user that is capable of being perceived by the user via the AR system:

determining a group of objects in the FOR of the user;

determining a field of view (FOV) of the user, the FOV comprising a portion of the FOR that is capable of being perceived at a given time by the user via the AR system;

identifying a subgroup of the group of objects that are located inside of the FOR of the user but outside of the FOV of the user, the subgroup including at least one interactable virtual object; and

displaying a visual aura for one or more of the objects in the subgroup of the group of objects such that at least a portion of the visual aura is perceivable in the FOV of the user,

wherein the visual aura represents contextual information associated with the at least one interactable virtual object in the subgroup;

accessing the contextual information associated with the at least one interactable object, and

dynamically changing the visual aura when the contextual information changes, wherein

the contextual information comprises location of the object and wherein the change in the contextual information comprises change in proximity to the FOV and the dynamically changing the visual aura comprises the aura thinning and further comprising, for at least one virtual object in the subgroup of virtual objects, informing the user of the at least one virtual object using a tactile effect.

31. The method of claim 30 , wherein the FOV is determined based at least partly on a pose of the user, and wherein the pose of the user comprises at least one of the following: head pose, eye pose, and body pose.

32. The method of claim 30 , further comprising: determining a light condition of the environment of the user; simulating optical effects of the light condition for one or more virtual objects in the group of the objects; and determining a placement of the visual aura associated with the one or more virtual objects based at least partly on the simulated optical effects.

33. The method of claim 32 , wherein the environment is one or more of virtual environment or physical environment.

34. The method of claim 32 , wherein the placement of the aura comprises one or more of the following: brightness, position, shape, or size and the size indicates an urgency of the virtual object.

35. The method of claim 32 , wherein determining a placement of the visual aura further comprises: identify a virtual object in the subgroup of the group of objects which are in the FOR of the user but are outside of the FOV of the user; and collinearly align the imaging system and at least one eye of the user with an interior edge of a visual aura associated with the virtual object.

36. The method of claim 30 , further comprising: updating the subgroup of the group of objects based at least partly on a change of the pose of the user; and updating the first or second representation of the visual aura based on the updated subgroup of the group of objects.

37. A method for providing an indication of an object in an environment of a user, the method comprising: under control of an augmented reality (AR) system comprising an imaging system, the AR system configured to permit user interaction with virtual objects in a field of regard (FOR) of the user, the FOR comprising a portion of the environment around the user that is capable of being perceived by the user via the AR system:

determining a group of objects in the FOR of the user;

determining a field of view (FOV) of the user, the FOV comprising a portion of the FOR that is capable of being perceived at a given time by the user via the AR system;

identifying a subgroup of the group of objects that are located inside of the FOR of the user but outside of the FOV of the user, the subgroup including at least one interactable virtual object; and

displaying a visual aura for one or more of the objects in the subgroup of the group of objects such that at least a portion of the visual aura is perceivable in the FOV of the user, wherein the visual aura represents contextual information associated with the at least one interactable virtual object in said subgroup and further comprising:

accessing the contextual information associated with the at least one interactable virtual object, and

dynamically changing the visual aura when the contextual information changes, wherein

the AR system displays a first AR display for a first eye of the user and displays a second, different, AR display for a second eye of the user, and wherein displaying the visual aura on the edge of the FOV of the user comprises: displaying a first representation of the visual aura by the first AR display, wherein

the displaying the visual aura comprises displaying the visual aura on a virtual user interface (UI) and further comprising interacting with the virtual UI using a totem, to generate more virtual content on the virtual UI.

38. The method of claim 37 , further comprising displaying a second representation of the visual aura by the second AR display, the second representation different from the first representation.

39. The method of claim 38 , wherein the first representation of the visual aura and the second representation of the visual aura are rendered separately to each eye to match peripheral vision or avoid depth perception of the visual aura.

40. The method of claim 37 , wherein the contextual information comprises urgency of the object, the change in the contextual information comprises an increased urgency and the dynamically changing the visual aura comprises the aura brightening: or,

the contextual information comprises received messages, the change in the contextual information comprises an increased number of received messages and the dynamically changing the visual aura comprises the aura becoming thicker.

Assignments (5)

Continuity (4)

References Cited (173)