IP Library Granted Patent US 10,078,919
Granted Patent B2
US 10,078,919 · App. 15/473,444 · Granted Sep 18, 2018

Interactions with 3D virtual objects using poses and multiple-DOF controllers

Inventors: James M. Powderly (Ft. Lauderdale, FL); Savannah Niles (Ft. Lauderdale, FL); Frank Hamilton (Ft. Lauderdale, FL); Marshal A. Fontaine (Sunrise, FL); Paul Armistead Hoover (Bothell, WA)
Assignee: Magic Leap, Inc.
G06T19/006G06F3/011G06F3/0346G06F1/163
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,078,919
App. No.
15/473,444
Granted
Sep 18, 2018
Kind
B2
Abstract

A wearable system can comprise a display system configured to present virtual content in a three-dimensional space, a user input device configured to receive a user input, and one or more sensors configured to detect a user's pose. The wearable system can support various user interactions with objects in the user's environment based on contextual information. As an example, the wearable system can adjust the size of an aperture of a virtual cone during a cone cast (e.g., with the user's poses) based on the contextual information. As another example, the wearable system can adjust the amount of movement of virtual objects associated with an actuation of the user input device based on the contextual information.

Claims (45)

1. A system for interacting with objects for a wearable device, the system comprising:

a display system of a wearable device configured to present a three-dimensional (3D) view to a user and permit a user interaction with virtual objects in a field of regard (FOR) of a user, the FOR comprising a portion of the environment around the user that is capable of being perceived by the user via the display system;

a sensor configured to acquire data associated with a pose of the user;

a hardware processor in communication with the sensor and the display system, the hardware processor programmed to:

determine a pose of the user based on the data acquired by the sensor;

initiate a cone cast on a group of virtual objects in the FOR, the cone cast comprises casting a virtual cone with an aperture in a direction based at least partly on the pose of the user, wherein the aperture has a dynamically-adjustable size;

analyze contextual information associated with the group of virtual objects in the user's environment to calculate the dynamically-adjustable size of the aperture for the cone cast toward the group of virtual objects;

automatically update the dynamically-adjustable size of the aperture of the virtual cone based at least partly on the contextual information of the group of virtual objects; and

cause the display system to render a visual representation of the virtual cone for the cone cast.

2. The system of claim 1 , wherein the contextual information comprises at least one of: a type, a layout, a location, a size, or a density of a subgroup of virtual objects within the field of view (FOV) of the user, wherein the FOV comprises a portion of the FOR that is capable of being perceived at a given time by the user via the display system.

3. The system of claim 2 , wherein the density of the subgroup of virtual objects within the FOV of the user is calculated by at least one of:

calculating a number of virtual objects in the subgroup of virtual objects;

calculating a percentage of the FOV that is covered by the subgroup of virtual objects; or

calculating a contour map for virtual objects in the subgroup of virtual objects.

4. The system of claim 1 , wherein the hardware processor is further programmed to detect a collision between the virtual cone and one or more virtual objects within the group of virtual objects in the FOR, and wherein in response to detecting the collision, the hardware processor is further programmed to present a focus indicator to the one or more virtual objects.

5. The system of claim 4 , wherein the hardware processor is programmed to apply an occlusion disambiguation technique to the one or more virtual objects collided with the virtual cone to identify an occluded virtual object.

6. The system of claim 1 , wherein the cone comprises a central ray and wherein the aperture is transverse to the central ray.

7. The system of claim 1 , wherein the virtual cone comprises a proximal end and wherein the proximal end is anchored to at least one of the following locations: a location in-between the user's eyes, a location on a portion of a user's arm, a location on a user input device, or any other location in the environment of the user.

8. The system of claim 1 , wherein the hardware processor is further programmed to receive an indication from a user input device anchoring a depth of the virtual cone to a depth plane and wherein cone cast is performed on the group of virtual objects within the depth plane.

9. A method for interacting with objects for a wearable device, the method comprising:

under control of a wearable device comprising a display system and hardware processor:

receiving a selection of a target virtual object displayed to a user at a first position in a three-dimensional (3D) space;

receiving an indication to move the target virtual object from the first position, wherein the indication comprises a first amount of movement associated with the user in a first direction in the 3D space;

analyzing a distance between the target virtual object and the user in a second direction different from the first direction;

calculating a multiplier to map the first amount of movement associated with the user into a second amount of movement for the target virtual object, the multiplier calculated based at least partly on the distance between the target virtual object and the user in the second direction;

calculating the second amount of movement in the first direction for the target virtual object, the second amount of movement calculated based at least partly on the first amount of movement and the multiplier; and

causing the target virtual object to be displayed to the user at a second position, the second position based at least in part on the first position and the second amount of movement.

10. The method of claim 9 , wherein the multiplier increases proportionally with an increase in the distance between the target virtual object and the user.

11. The method of claim 9 , wherein the multiplier is applied to one or more of: a position change, a speed, or an acceleration, for calculating the second amount of movement.

12. The method of claim 9 , wherein the indication to move the target virtual object comprises at least one of: an actuation of a user input device associated with the wearable device or a change in a pose of the user.

13. The method of claim 12 , wherein the pose comprises one or more of: a head pose, an eye pose, or a body pose.

14. The method of claim 9 , wherein the multiplier is greater than one where the distance between the target virtual object and the user is beyond the user's hand reach.

15. A system for interacting with objects for a wearable device, the system comprising:

a display system of a wearable device configured to present a three-dimensional (3D) view to a user, the 3D view showing a target virtual object;

a hardware processor in communication with the display system, the hardware processor programmed to:

receive an indication to move the target virtual object from a first position, wherein the indication comprises a first amount of movement associated with a user in a target direction in the 3D space;

analyze a distance between the target virtual object and the user;

calculate a multiplier to map the first amount of movement associated with the user into a second amount of movement for the target virtual object, the multiplier calculated based at least partly on the distance between the target virtual object and the user;

calculate the second amount of movement in the target direction for the target virtual object, the second amount of movement calculated based at least partly on the first amount of movement and the multiplier; and

by cause the display system to display the target virtual object at a second position, the second position based at least in part on the first position and the second amount of movement.

16. The system of claim 15 , wherein the indication to move the target virtual object comprises a change in a pose of a user of the wearable device or an input received from a user input device associated with the wearable device.

17. The system of claim 15 , wherein the multiplier equals to one when the distance is less than a threshold distance, wherein the threshold distance equals to a hand reach of the user.

18. The system of claim 15 , wherein the multiplier increases proportionally with an increase in the distance between the target virtual object and the user.

19. The system of claim 15 , wherein the multiplier is applied to one or more of: a position change, a speed, or an acceleration, for calculating the second amount of movement.

20. The system of claim 15 , wherein the multiplier is greater than one where the distance between the target virtual object and the user is beyond the user's hand reach.

Assignments (6)
SECURITY INTEREST Recorded Oct 15, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073109/0476 →
ASSIGNMENT OF SECURITY INTEREST IN PATENTS Recorded Nov 7, 2019
From: JPMORGAN CHASE BANK, N.A.
To: CITIBANK, N.A.
Reel/Frame 050967/0138 →
PATENT SECURITY AGREEMENT Recorded Aug 22, 2019
From: MAGIC LEAP, INC.; MOLECULAR IMPRINTS, INC.; MENTOR ACQUISITION ONE, LLC
To: JP MORGAN CHASE BANK, N.A.
Reel/Frame 050138/0287 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 16, 2018
From: POWDERLY, JAMES M.; NILES, SAVANNAH; HAMILTON, FRANK; FONTAINE, MARSHAL A.
To: MAGIC LEAP, INC.
Reel/Frame 045825/0015 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 16, 2018
From: HOOVER, PAUL ARMISTEAD
To: GIRLING KELLY DESIGN GROUP, LLC D.B.A. ARTEFACT
Reel/Frame 045825/0047 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 16, 2018
From: GIRLING KELLY DESIGN GROUP, LLC D.B.A. ARTEFACT
To: MAGIC LEAP, INC.
Reel/Frame 045825/0503 →
Continuity (3)
Provisional Application 62325679 · Apr 21, 2016
Provisional Application 62316030 · Mar 31, 2016
Related Publication 20170287225A1 · Oct 5, 2017
Cited By (6)
US 12,406,454 US 12,444,146 US 12,449,950 US 12,474,788 US 12,591,297 US 12,651,420