Granted Patent B2

US 11,328,484 · App. 15/599,162 · Granted May 10, 2022

Inventors: James M. Powderly (Ft. Lauderdale, FL); Alysha Naples (Ft. Lauderdale, FL); Paul Armistead Hoover (Bothell, WA); Tucker Spofford (Seattle, WA)

Assignee: Magic Leap, Inc.

G06T19/006G02B27/017G06F3/011G06F3/012G06F3/013G06F3/015G06F3/0482G06F3/04815G06K7/1408G06K9/00375G06K19/06009G06F3/017

Abstract

Claims (79)

1. A method for generating virtual content in a three-dimensional (3D) physical environment of a user, the method comprising:

under control of a hardware processor:

analyzing data acquired from a pose sensor to identify a pose of a user;

identifying a physical surface in a 3D physical environment of the user based at least partly on the pose;

receiving an indication to initiate an interaction with the physical surface;

determining an orientation of the physical surface;

accessing information indicating a first set of virtual objects associated with a first surface orientation and a second set of virtual objects associated with a second surface orientation;

in response to determining that the orientation of the physical surface is the first surface orientation, selecting the first set of virtual objects;

in response to determining that the orientation of the physical surface is the second surface orientation, selecting the second set of virtual objects;

determining a set of available user interface options corresponding to the selected first or second set of virtual objects;

determining a first number of user interface options from the set of available user interface options that can legibly fit in a virtual user interface associated with an area available on the physical surface, wherein the first number is less than a number of user interface options in the set of available user interface options;

selecting, from the set of available user interface options, a subset of user interface options equal to the first number;

generating the virtual user interface including the subset of user interface options;

generating an instruction for presenting the virtual user interface containing the subset of user interface options that can legibly fit in the virtual user interface in a 3D view to the user; and

in response to selection of a first user interface option of the subset of user interface options, generating an instruction for displaying a corresponding first virtual object on the physical surface.

2. The method of claim 1 , wherein the pose comprises at least one of: an eye gaze, a head pose, or a gesture.

3. The method of claim 1 , wherein identifying the physical surface comprises performing a conecasting based on the pose.

4. The method of claim 1 , wherein the indication to initiate the interaction with the physical surface comprises an actuation of a user input device or a change in the pose of the user.

5. The method of claim 1 , further comprising receiving physiological parameters of the user; and determining a psychological state of the user, wherein said determining the set of available user interface options is based at least partly on the psychological state of the user.

6. The method of claim 5 , wherein physiological parameters are related to at least one of: a heart rate, a pupil dilation, a galvanic skin response, a blood pressure, an encephalographic state, a respiration rate, or an eye movement.

7. The method of claim 1 , wherein generating the instruction for presenting the subset of user interface options in the 3D view to the user comprises:

generating a virtual menu comprising the subset of user interface options;

determining a spatial location of the virtual menu based on a characteristic of the physical surface; and

generating a display instruction for presentation of the virtual menu at the spatial location in the 3D physical environment of the user.

8. The method of claim 1 , wherein the set of available user interface options is further determined based on user information comprises one or more of: user age, user mood, user preference, information associated with the user, a physiological parameter of the user, a psychological state of the user, an emotional state of the user, a mood of a group of users, or a user characteristic.

9. The method of claim 1 , wherein the virtual user interface comprises a virtual menu.

10. A system for generating virtual content in a physical environment of a user, the system comprising:

a display system of a wearable device configured to present virtual content; and

a hardware processor in communication with the display system, the hardware processor programmed to:

determine a pose of a user based on data acquired from a pose sensor;

identify a physical surface in a 3D physical environment of the user based at least partly on the pose;

receive an indication to initiate an interaction with the physical surface;

determine an orientation of the physical surface;

access information indicating a first set of virtual objects associated with a first surface orientation and a second set of virtual objects associated with a second surface orientation;

in response to determining that the orientation of the physical surface is the first surface orientation, select the first set of virtual objects;

in response to determining that the orientation of the physical surface is the second surface orientation, select the second set of virtual objects;

determine a set of available user interface options corresponding to the selected first or second set of virtual objects;

determine a first number of user interface options from the set of available user interface options that can legibly fit in a virtual user interface associated with an area available on the physical surface, wherein the first number is less than a number of user interface options in the set of available user interface options;

select, from the set of available user interface options, a subset of user interface options equal to the first number;

generate the virtual user interface including the subset of user interface options;

generate an instruction for presenting the virtual user interface containing the subset of user interface options that can legibly fit in the virtual user interface in a 3D view to the user; and

in response to selection of a first user interface option of the subset of user interface options, generate an instruction for displaying a corresponding first virtual object on the physical surface.

11. The system of claim 10 , wherein the pose comprises at least one of: an eye gaze, a head pose, or a gesture.

12. The system of claim 10 , wherein to identify the physical surface, the hardware processor is programmed to perform a conecasting based on the pose.

13. The system of claim 10 , wherein the indication to initiate the interaction with the physical surface comprises an actuation of a user input device or a change in the pose of the user.

14. The system of claim 10 , further comprising receiving the user's physiological parameters; wherein said determining the set of available user interface options is based at least partly on the psychological state of the user.

15. The system of claim 14 , wherein the user's physiological parameters are related to at least one of: a heart rate, a pupil dilation, a galvanic skin response, a blood pressure, an encephalographic state, a respiration rate, or an eye movement.

16. The system of claim 10 , wherein generating the instruction for presenting the subset of user interface options to the user comprises:

generating a virtual menu comprising the subset of user interface options;

determining a spatial location of the virtual menu based on a characteristic of the physical surface; and

generating a display instruction for presentation of the virtual menu at the spatial location in the physical environment of the user.

17. The system of claim 10 , wherein the set of available user interface options is further determined based on user information comprises one or more of: user age, user mood, user preference, information associated with the user, a physiological parameter of the user, a psychological state of the user, an emotional state of the user, a mood of a group of users, or a user characteristic.

18. The system of claim 10 , wherein the virtual user interface comprises a virtual menu.

19. A method for generating virtual content in a three-dimensional (3D) physical environment of a user, the method comprising:

under control of a hardware processor:

analyzing data acquired from a pose sensor to identify a pose of a user;

identifying a wall in a 3D physical environment of the user based at least partly on the pose;

accessing information indicating that a first set of virtual objects is associated with a first surface orientation and that a second set of virtual objects associated with a second surface orientation;

receiving an indication to initiate an interaction with the wall;

in response to determining that the first surface orientation is horizontal, select the first set of virtual objects;

determining user information associated with an area available on the wall;

accessing a set of available user interface options associated with the first set of virtual objects and based on the user information;

determining, based at least on the area available on the wall, a first set of user interface options from the set of available user interface options that can legibly fit in a virtual user interface associated with the area available on the wall, wherein a number of the first set of user interface options is less than a number of user interface options in the set of available user interface options;

generating the virtual user interface including the first set of user interface options; and

generating an instruction for presenting the virtual user interface containing the first set of user interface options that can legibly fit in the virtual user interface in a 3D view to the user.

20. The method of claim 19 , wherein the virtual user interface comprises a virtual menu.

21. A system for generating virtual content in a physical environment of a user, the system comprising:

a display system of a wearable device configured to present virtual content; and

a hardware processor in communication with the display system, the hardware processor programmed to perform operations comprising:

analyzing data acquired from a pose sensor to identify a pose of a user;

identifying a wall in a 3D physical environment of the user based at least partly on the pose;

accessing information indicating that a first set of virtual objects is associated with a first surface orientation and that a second set of virtual objects associated with a second surface orientation;

receiving an indication to initiate an interaction with the wall;

in response to determining that the first surface orientation is horizontal, select the first set of virtual objects;

determining user information associated with an area available on the wall;

accessing a set of available user interface options associated with the first set of virtual objects and based on the user information;

determining, based at least on the area available on the wall and on the first surface orientation of the wall, a first set of user interface options from the set of available user interface options that can legibly fit in a virtual user interface associated with an area available on the wall, wherein a number of the first set of user interface options is less than a number of user interface options in the set of available user interface options;

generating the virtual user interface including the first set of user interface options; and

generating an instruction for presenting the virtual user interface containing the first set of user interface options that can legibly fit in the virtual user interface in a 3D view to the user.

Assignments (8)