IP Library Granted Patent US 10,417,831
Granted Patent B2
US 10,417,831 · App. 16/053,620 · Granted Sep 17, 2019

Interactions with 3D virtual objects using poses and multiple-DOF controllers

Inventors: James M. Powderly (Ft. Lauderdale, FL); Savannah Niles (Ft. Lauderdale, FL); Frank Hamilton (Ft. Lauderdale, FL); Marshal A. Fontaine (Sunrise, FL); Paul Armistead Hoover (Bothell, WA)
Assignee: Magic Leap, Inc.
G06T19/006G06F3/011G06F3/0346G06F1/163
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,417,831
App. No.
16/053,620
Granted
Sep 17, 2019
Kind
B2
Abstract

A wearable system can comprise a display system configured to present virtual content in a three-dimensional space, a user input device configured to receive a user input, and one or more sensors configured to detect a user's pose. The wearable system can support various user interactions with objects in the user's environment based on contextual information. As an example, the wearable system can adjust the size of an aperture of a virtual cone during a cone cast (e.g., with the user's poses) based on the contextual information. As another example, the wearable system can adjust the amount of movement of virtual objects associated with an actuation of the user input device based on the contextual information.

Claims (40)

1. A system comprising:

a display system of a wearable device configured to present a three-dimensional (3D) view to a user and permit a user interaction with objects in a field of regard (FOR) of a user, the FOR comprising a portion of the environment around the user that is capable of being perceived by the user via the display system;

a sensor configured to acquire data associated with a pose of the user;

a hardware processor in communication with the sensor and the display system, the hardware processor programmed to:

initiate a cone cast, wherein the cone cast comprises a cast of a virtual cone with a dynamically-adjustable aperture;

identify one or more contextual features in the environment;

dynamically resize the aperture of the virtual cone based at least in part on the one or more contextual features;

translate the virtual cone based on the data associated with the pose of the user;

scan for a collision between the virtual cone and one or more objects in the environment;

identify, in response to the collision, a collided object; and

perform a user interaction associated with the collided object.

2. The system of claim 1 , wherein the contextual features comprise at least one of: a type, a layout, a location, a size, a distance, or a density of objects within a field of view (FOV) of the user, wherein the FOV comprises a portion of the FOR that is capable of being perceived at a given time by the user via the display system.

3. The system of claim 2 , wherein to calculate the density of objects within the FOV of the user, the hardware processor is programmed to:

calculate a number of objects in the FOV;

calculate a fraction of the FOV that is covered by the objects; or

calculate a contour map for the objects.

4. The system of claim 1 , wherein the contextual features comprise at least one of: a preference of the user, a physical condition associated with the user, or information associated with the environment.

5. The system of claim 1 , wherein to dynamically resize the aperture of the virtual cone, the hardware processor is programmed to resize the aperture between a minimum size and a maximum size.

6. The system of claim 5 , wherein the minimum size is zero.

7. The system of claim 1 , wherein the hardware processor is further programmed to render a visualization of the virtual cone.

8. The system of claim 1 , wherein the hardware processor is further programmed to render a focus indicator associated with the collided object.

9. The system of claim 1 , wherein to identify the collided object, the hardware processor is programmed to determine that the collided object intersects with a virtual surface of the virtual cone or falls within the virtual cone.

10. The system of claim 1 , wherein the collided object comprises multiple collided objects, and the hardware processor is programmed to apply an occlusion disambiguation technique to the multiple collided objects to identify an occluded object or to determine a depth ordering or position among occluded objects.

11. The system of claim 1 , wherein the collided object comprises multiple collided objects, and the hardware processor is programmed to present a user interface element to select one or more of the multiple collided objects.

12. The system of claim 1 , wherein the virtual cone comprises a central ray, the aperture is transverse to the central ray, and a direction of the central ray is based on the pose of the user.

13. The system of claim 1 , wherein the virtual cone comprises a proximal end, and the hardware processor is programmed to anchor the proximal end to at least one of the following locations: a location in-between the user's eyes, a location on a portion of a user's arm, or a location on a user input device.

14. The system of claim 1 , wherein the virtual cone comprises a distal end, and the hardware processor is programmed to scan for collisions with one or more objects within the distal end of the virtual cone.

15. The system of claim 1 , wherein the virtual cone comprises a proximal end, a distal end, or a depth, and the hardware processor is programmed to anchor the proximal end, the distal end, or the depth of the virtual cone based on: the contextual features, user input, a body gesture, a body pose, a direction of gaze, or a voice command.

16. The system of claim 1 , wherein the user interaction with the collided object comprises one or more of:

a selection of the collided object;

a movement of the collided object;

display of a menu or toolbar associated with the collided object; or

a game operation on a virtual avatar in a game.

17. The system of claim 1 , wherein to translate the virtual cone based on the data associated with the pose of the user, the hardware processor is programmed to apply a multiplier to the pose of the user.

18. The system of claim 17 , wherein the multiplier increases with distance from the user.

19. The system of claim 1 , wherein to dynamically resize the aperture of the virtual cone, the hardware processor is programmed to increase a size of the aperture at a distal end of the virtual cone.

20. The system of claim 1 , wherein to initiate the cone cast, the hardware processor is programmed to extend a distal end of the virtual cone until the distal end reaches a termination threshold.

21. The system of claim 20 , wherein the termination threshold comprises a threshold distance or a boundary of the environment.

22. The system of claim 1 , wherein the virtual cone has a depth, and the hardware processor is programmed to scan for a collision within the depth of the virtual cone.

23. The system of claim 1 , wherein the objects comprise virtual objects.

Assignments (6)
SECURITY INTEREST Recorded Oct 15, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073109/0476 →
ASSIGNMENT OF SECURITY INTEREST IN PATENTS Recorded Nov 7, 2019
From: JPMORGAN CHASE BANK, N.A.
To: CITIBANK, N.A.
Reel/Frame 050967/0138 →
PATENT SECURITY AGREEMENT Recorded Aug 22, 2019
From: MAGIC LEAP, INC.; MOLECULAR IMPRINTS, INC.; MENTOR ACQUISITION ONE, LLC
To: JP MORGAN CHASE BANK, N.A.
Reel/Frame 050138/0287 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 21, 2019
From: HOOVER, PAUL ARMISTEAD
To: GIRLING KELLY DESIGN GROUP, LLC D.B.A. ARTEFACT
Reel/Frame 049246/0192 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 21, 2019
From: GIRLING KELLY DESIGN GROUP, LLC D.B.A. ARTEFACT
To: MAGIC LEAP, INC.
Reel/Frame 049246/0219 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 21, 2019
From: POWDERLY, JAMES M.; NILES, SAVANNAH; HAMILTON, FRANK; FONTAINE, MARSHAL A.
To: MAGIC LEAP, INC.
Reel/Frame 049246/0255 →
Continuity (4)
Continuation 15473444 · Mar 29, 2017
Provisional Application 62316030 · Mar 31, 2016
Provisional Application 62325679 · Apr 21, 2016
Related Publication 20180365901A1 · Dec 20, 2018
Cited By (3)
US 12,406,454 US 12,474,788 US 12,591,297