IP Library Granted Patent US 11,733,786
Granted Patent B2
US 11,733,786 · App. 17/981,311 · Granted Aug 22, 2023

Selecting virtual objects in a three-dimensional space

Inventors: James M. Powderly (Ft. Lauderdale, FL); Savannah Niles (Ft. Lauderdale, FL); Frank Hamilton (Martinsburg, WV); Marshal A. Fontaine (St. Augustine, FL); Rony Abovitz (Weston, FL); Alysha Naples (London, GB)
Assignee: Magic Leap, Inc.
G06F3/0346G06F1/163G06F3/011G06F3/012G06F3/013G06F3/016G06F3/017G06F3/0482G06F3/04815G06F3/04883
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,733,786
App. No.
17/981,311
Granted
Aug 22, 2023
Kind
B2
Abstract

Systems and methods for interacting with virtual objects in a three-dimensional space using a wearable system are disclosed. The wearable system can be programmed to permit user interaction with interactable objects in a field of regard (FOR) of a user. The FOR includes a portion of the environment around the user that is capable of being perceived by the user via the AR system. The system can determine a group of interactable objects in the FOR of the user and determine a pose of the user. The system can update, based on a change in the pose or a field of view (FOV) of the user, a subgroup of the interactable objects that are located in the FOV of the user and receive a selection of a target interactable object from the subgroup of interactable objects. The system can initiate a selection event on the target interactable object.

Claims (30)

1. A wearable system for selecting a virtual object located in a three-dimensional (3D) space, the wearable system comprising: a display system configured to present virtual objects in a 3D space; a non-transitory data store configured to store interactable objects in the 3D space; a sensor configured to determine a pose of a user; and a hardware processor programmed to communicate with the display system, the data store, and the sensor to: determine the pose of the user based on data received from the sensor; determine a field of view (FOV) of the user based on the pose of the user, the FOV comprising a portion of an environment of the user that is perceived at a given time by the user; identify a group of interactable objects in the FOV; identify a target interactable object in the FOV based least partly on the pose of the user; initiate a selection event associated with the target interactable object; and in response to initiating a selection event on the target interactable object, present, in a two-dimensional (2D) interface, virtual objects within a threshold range of the target interactable object in the 3D space.

2. The system of claim 1 , wherein the hardware processor is programmed further to index the group of interactable objects based on a coordinate associated with a distance from a position of the target interactable object.

3. The system of claim 2 , wherein the coordinate comprises a lateral distance, vertical distance, and depth distance relative to the position of the target interactable object.

4. The system of claim 2 , wherein the hardware processor is programmed further to determine the position of the target interactable object based on the user's pose.

5. The system of claim 1 , wherein the group of interactable objects is stored in a data structure including an index associated with each interactable object, each index being determined based on a position of the interactable object in the 3D space.

6. The system of claim 1 , wherein the 2D interface is interactable via a user input device.

7. The system of claim 1 , wherein to identify the target interactable object in the FOV, the processor is configured to perform at least one of:

determine a path of a gaze of the user based on the pose of the user and select an interactable object which intersects the path of the gaze as the target interactable object; or select a leftmost or a rightmost interactable object in the user's FOV as the target interactable object, wherein the leftmost or the rightmost interactable object is selected based on indices associated with the group of interactable objects.

8. The system of claim 1 , wherein the processor is configured to initiate the selection event in response to receiving an input from a user input device or detecting a change of the pose of the user.

9. The system of claim 1 , wherein the processor is further configured to present a focus indicator associated with the target interactable object.

10. The system of claim 1 , further comprising a geolocation sensor configured to acquire data associated with a location of the user, and wherein the target interactable object comprises a weather application which is programmed to:

determine the location of the user based on the data acquired by the geolocation sensor;

communicate with a remote computing device to acquire weather data based on the location of the user;

generate a virtual element associated with the weather data; and

superimpose the virtual element in the 3D space of the user.

11. A method for selecting a virtual object located in a three-dimensional (3D) space, the method comprising: under control of a hardware processor programmed to communicate with a display system, a data store, and a sensor configured to determine a pose of a user: determining the pose of the user based on data received from the sensor; determining a field of view (FOV) of the user based on the pose of the user, the FOV comprising a portion of an environment of the user that is perceived at a given time by the user; identifying a group of interactable objects in the FOV; identifying a target interactable object in the FOV based least partly on the pose of the user; initiating a selection event associated with the target interactable object; and in response to initiating a selection event on the target interactable object, presenting, in a two-dimensional (2D) interface, virtual objects within a threshold range of the target interactable object in a 3D space.

12. The method of claim 11 , further comprising indexing the group of interactable objects based on a coordinate associated with a distance from a position of the target interactable object.

13. The method of claim 12 , wherein the coordinate comprises a lateral distance, vertical distance, and depth distance relative to the position of the target interactable object.

14. The method of claim 12 , further comprising determining the position of the target interactable object based on the user's pose.

15. The method of claim 11 , further comprising indexing each interactable object based on a position of the target interactable object in the 3D space.

16. The method of claim 11 , further comprising storing the group of interactable objects in a data structure including an index associated with each interactable object, each index being determined based on a position of the interactable object in the 3D space.

17. The method of claim 11 , further comprising receiving user interaction with the 2D interface via a user input device.

18. The method of claim 11 , further comprising at least one of: determining a path of a gaze of the user based on the pose of the user and select an interactable object which intersects the path of the gaze as the target interactable object; or selecting a leftmost or a rightmost interactable object in the user's FOV as the target interactable object, wherein the leftmost or the rightmost interactable object is selected based on indices associated with the group of interactable objects.

19. The method of claim 11 , further comprising initiating the selection event in response to receiving an input from a user input device or detecting a change of the pose of the user.

20. The method of claim 11 , further comprising:

using a geolocation sensor, acquiring data associated with a location of the user;

determining the location of the user based on the data acquired by the geolocation sensor;

communicating with a remote computing device to acquire weather data based on the location of the user;

generating a virtual element associated with the weather data; and

superimposing the virtual element in the 3D space of the user.

Assignments (3)
SECURITY INTEREST Recorded Oct 20, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073008/0696 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 6, 2023
From: POWDERLY, JAMES M.; NILES, SAVANNAH; HAMILTON, FRANK; FONTAINE, MARSHAL A.; ABOVITZ, RONY; NAPLES, ALYSHA
To: MAGIC LEAP, INC.
Reel/Frame 063244/0802 →
SECURITY INTEREST Recorded Feb 7, 2023
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 062681/0065 →
Continuity (7)
Continuation 17454793 · Nov 12, 2021
Continuation 16682794 · Nov 13, 2019
Continuation 15296869 · Oct 18, 2016
Provisional Application 62316179 · Mar 31, 2016
Provisional Application 62301422 · Feb 29, 2016
Provisional Application 62244115 · Oct 20, 2015
Related Publication 20230057175A1 · Feb 23, 2023