IP Library Granted Patent US 11,175,750
Granted Patent B2
US 11,175,750 · App. 16/682,794 · Granted Nov 16, 2021

Selecting virtual objects in a three-dimensional space

Inventors: James M. Powderly (Ft. Lauderdale, FL); Savannah Niles (Ft. Lauderdale, FL); Frank Hamilton (Plantation, FL); Marshal A. Fontaine (St. Augustine, FL); Rony Abovitz (Weston, FL); Alysha Naples (London, GB)
Assignee: Magic Leap, Inc.
G06F3/0346G06F1/163G06F3/011G06F3/012G06F3/013G06F3/016G06F3/017G06F3/0482G06F3/04815G06F3/04883
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,175,750
App. No.
16/682,794
Granted
Nov 16, 2021
Kind
B2
Abstract

Systems and methods for interacting with virtual objects in a three-dimensional space using a wearable system are disclosed. The wearable system can be programmed to permit user interaction with interactable objects in a field of regard (FOR) of a user. The FOR includes a portion of the environment around the user that is capable of being perceived by the user via the AR system. The system can determine a group of interactable objects in the FOR of the user and determine a pose of the user. The system can update, based on a change in the pose or a field of view (FOV) of the user, a subgroup of the interactable objects that are located in the FOV of the user and receive a selection of a target interactable object from the subgroup of interactable objects. The system can initiate a selection event on the target interactable object.

Claims (73)

1. A method for selecting a virtual object located in three-dimensional (3D) space, the method comprising:

under control of an augmented reality (AR) system comprising computer hardware, the AR system configured to permit user interaction with interactable objects in a field of regard (FOR) of a user, the FOR comprising a portion of an environment around the user that is capable of being perceived by the user via the AR system:

determining a group of interactable objects in the FOR of the user;

determining a pose of the user;

determining a field of view (FOV) of the user based at least partly on the pose of the user, the FOV comprising a portion of the FOR that is perceived at a given time by the user;

updating, based on a change in the pose or the FOV of the user, a subgroup of the interactable objects that are located in the FOV of the user;

magnifying a display of the subgroup of interactable objects;

receiving a selection of a target interactable object from the subgroup of interactable objects; and

initiating a selection event on the target interactable object.

2. The method of claim 1 , further comprising storing the group of interactable objects in a data structure.

3. The method of claim 2 , wherein each interactable object in the group of interactable objects is represented in the data structure based at least in part on a location of the interactable object in the FOV of the user.

4. The method of claim 3 , wherein the location comprises a distance from an edge of the FOV of the user.

5. The method of claim 1 , wherein receiving the selection of the target interactable object from the subgroup of interactable objects comprises:

receiving a first input from a user device; and

in response to the first input, identifying the target interactable object from the subgroup of the interactable objects.

6. The method of claim 5 , further comprising:

receiving a second input from the user device; and

in response to the second input, initiating an interaction event on the target interactable object.

7. The method of claim 6 , wherein the target interactable object is the interactable object in the subgroup of interactable objects that is closest to a midpoint of the user's FOV.

8. The method of claim 6 , wherein the target interactable object is a leftmost or a rightmost interactable object in the subgroup of interactable objects in the user's FOV.

9. The method of claim 6 , wherein initiating the interaction event comprises one or more of the following:

resizing the target interactable object;

displaying a menu of the target interactable object;

browsing the menu of the target interactable object;

selecting a first item on the menu; searching for a second item in a database;

playing a video game associated with the target interactable object;

watching a video; or

conducting a teleconference.

10. The method of claim 1 , wherein receiving the selection of the target interactable object is performed by:

determining a path of the user's gaze based on the pose of the user; and

selecting an object which intersects the path of the user's gaze as the target interactable object.

11. The method of claim 1 , further comprising assigning a visible focus indicator to the target interactable object.

12. The method of claim 11 , wherein the visible focus indicator comprises at least one of a highlight, a halo aura, a color change, a size change, or a change in a perceived depth of the target interactable object.

13. The method of claim 1 , wherein initiating the selection event comprises one or more of the following:

changing the target interactable object to a different interactable object that becomes the target interactable object;

opening a menu associated with the target interactable object; or

receiving a confirmation from the user to select the target interactable object.

14. The method of claim 1 , wherein the group of interactable objects comprises virtual objects.

15. The method of claim 1 , wherein the pose of the user comprises an eye pose or a head pose.

16. The method of claim 1 , wherein receiving the selection of the target interactable object from the subgroup of interactable objects comprises receiving the selection using an eye gesture or a hand gesture.

17. The method of claim 1 , wherein receiving the selection of the target interactable object from the subgroup of interactable objects comprises receiving the selection recursively.

18. An augmented reality (AR) system for selecting a virtual object located in three-dimensional (3D) space, the system comprising:

a display system;

a network interface;

computer processors configured to communicate with the network interface and the display system to:

determine a group of interactable objects in the field of regard (FOR) of a user;

determine a pose of the user;

determine a field of view (FOV) of the user based at least partly on the pose of the user, the FOV comprising a portion of the FOR that is perceived at a given time by the user;

update, based on a change in the pose or the FOV of the user, a subgroup of the interactable objects that are located in the FOV of the user;

magnify a display of the subgroup of interactable objects;

receive a selection of a target interactable object from the subgroup of interactable objects; and

initiate a selection event on the target interactable object.

19. The system of claim 18 , wherein the computer processors are further configured to store the group of interactable objects in a data structure.

20. The system of claim 19 , wherein one or more interactable objects in the group of interactable objects is represented in the data structure based at least in part on a location of the interactable object in the FOV of the user.

21. The system of claim 20 , wherein the location comprises a distance from an edge of the FOV of the user.

22. The system of claim 18 , wherein receiving the selection of the target interactable object from the subgroup of interactable objects comprises:

receiving a first input from a user device; and

in response to the first input, identifying the target interactable object from the subgroup of the interactable objects.

23. The system of claim 18 , wherein the target interactable object is the interactable object in the subgroup of interactable objects that is closest to a midpoint of the user's FOV.

24. The system of claim 18 , wherein the target interactable object is a leftmost or a rightmost interactable object in the subgroup of interactable objects in the user's FOV.

25. The system of claim 18 , wherein receiving the selection of the target interactable object from the subgroup of interactable objects comprises:

determining a path of the user's gaze based on the pose of the user; and

selecting an object which intersects the path of the user's gaze as the target interactable object.

26. The system of claim 18 , wherein the computer processors are further configured to assign a visible focus indicator to the target interactable object.

27. The system of claim 26 , wherein the visible focus indicator comprises a highlight, a halo aura, a color change, a size change, or a change in a perceived depth of the target interactable object.

28. The system of claim 18 , wherein initiating a selection event comprises one or more of the following:

changing the target interactable object to a different interactable object that becomes the target interactable object;

opening a menu associated with the target interactable object; or

receiving a confirmation from user to select the target interactable object.

29. The system of claim 18 , wherein the group of interactable objects comprises virtual objects.

30. The system of claim 18 , wherein the pose of the user comprises an eye pose or a head pose.

31. The system of claim 18 , wherein the processors are configured to receive the selection of the target interactable object using an eye gesture or a hand gesture.

32. The system of claim 18 , wherein the processors are configured to receiving the selection of the target interactable object recursively.

Assignments (3)
SECURITY INTEREST Recorded May 24, 2022
From: MOLECULAR IMPRINTS, INC.; MENTOR ACQUISITION ONE, LLC; MAGIC LEAP, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 060338/0665 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 30, 2021
From: POWDERLY, JAMES M.; NILES, SAVANNAH; HAMILTON, FRANK; FONTAINE, MARSHAL A.; ABOVITZ, RONY; NAPLES, ALYSHA
To: MAGIC LEAP, INC.
Reel/Frame 056725/0523 →
SECURITY INTEREST Recorded May 21, 2020
From: MAGIC LEAP, INC.; MOLECULAR IMPRINTS, INC.; MENTOR ACQUISITION ONE, LLC
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 052729/0791 →
Cited By (3)
US 12,406,454 US 12,474,788 US 12,591,297