Granted Patent B2

US 12,591,297 · App. 18/596,054 · Granted Mar 31, 2026

Inventors: James M. Powderly (Fort Lauderdale, FL); Savannah Niles (Fort Lauderdale, FL); Jennifer M.R. Devine (Plantation, FL); Adam C. Carlson (Miami, FL); Jeffrey Scott Sommers (Mountain View, CA); Praveen Babu J D (Plantation, FL); Ajoy Savio Fernandes (Plantation, FL); Anthony Robert Sheeder (Fort Lauderdale, FL)

Assignee: MAGIC LEAP, INC.

G06F3/011G02B6/0076G02B27/0172G02B27/0179G06F1/163G06F3/013G06F3/017G06F3/167G06T19/003G06T19/006G02B2027/0127G02B2027/0138G02B2027/014G02B2027/0187G06F2203/0381

Abstract

Claims (40)

1 . A method for interacting with virtual content, comprising:

receiving a first direct user input corresponding to a first virtual object and associating the first direct user input with a first weight;

determining a first confidence score correlated to the first direct user input, wherein the first confidence score is based, at least in part, on a first authentication level associated with the first direct user input;

receiving a second direct user input, corresponding to a second virtual object and having a different modality than the first direct user input, and associating the second direct user input with a second weight;

determining a second confidence score correlated to the second direct user input, wherein the second confidence score is based, at least in part, on a second authentication level associated with the second direct user input, the second authentication level being different than the first authentication level;

calculating an aggregate confidence score from at least the first confidence score, the first weight, the second confidence score, and the second weight;

identifying a subject virtual object from at least the first virtual object and the second virtual object based on the aggregate confidence score; and

manipulating the subject virtual object to meet a parameter condition.

2 . The method of claim 1 , wherein the first direct user input is selected from a group consisting of speech input, hand gesture, head pose, and eye gaze.

3 . The method of claim 1 , wherein the second direct user input is selected from a group consisting of speech input, hand gesture, head pose, and eye gaze.

4 . The method of claim 1 , wherein the parameter condition is a position of the subject virtual object.

5 . The method of claim 1 , wherein the parameter condition is a size of the subject virtual object.

6 . The method of claim 1 , wherein the parameter condition is a functionality of the subject virtual object.

7 . The method of claim 1 , wherein the aggregate confidence score is based, at least in part, on a lattice compression logic function.

8 . A method for interacting with virtual content, comprising:

receiving a direct user input corresponding to a first virtual object and associating the direct user input with a first weight;

determining a first confidence score correlated to the direct user input;

receiving an indirect user input corresponding to a second virtual object and associating the indirect user input with a second weight;

determining a second confidence score correlated to the indirect user input;

calculating an aggregate confidence score from at least the first confidence score, the first weight, the second confidence score, and the second weight;

identifying a subject virtual object from at least the first virtual object and the second virtual object based on the aggregate confidence score; and

manipulating the subject virtual object to meet a parameter condition.

9 . The method of claim 8 , wherein the direct user input is selected from a group consisting of speech input, hand gesture, head pose, and eye gaze.

10 . The method of claim 9 , wherein the indirect user input is geolocation.

11 . The method of claim 8 , wherein the parameter condition is at least one of a position of the subject virtual object, a size of the subject virtual object, or a functionality of the subject virtual object.

12 . A system for interacting with objects for a wearable device, the system comprising:

a display system of a wearable device configured to present a three-dimensional (3D) view to a user and permit a user interaction with objects in a field of regard (FOR) of the user, the FOR comprising a portion of an environment around the user that is capable of being perceived by the user via the display system;

a hardware processor in communication with a sensor and the display system, the hardware processor programmed to:

receive multimodal inputs for the user interaction;

parse the multimodal inputs to identify a subject, a command, and a parameter of the user interaction; and

execute the user interaction based on the subject, the command, and the parameter;

wherein to identify the subject of the user interaction, the wearable device is programmed to: access a first value to a first input of the multimodal inputs; access a second value to a second input of the multimodal inputs; calculate a first confidence score for a first candidate virtual object based on the first value or the second value; calculate a second confidence score for a second candidate virtual object based on the first value or the second value; and set the subject to be either the first candidate virtual object or the second candidate virtual object based on a comparison of the first confidence score and the second confidence score such that the subject is the first candidate virtual object or the second candidate virtual object with a higher confidence score,

wherein the first input is associated with a first weight and the second input is associated with a second weight, and wherein the first value and the second value are assigned based on the first weight and the second weight respectively.

13 . The system of claim 12 , wherein the system comprises at least one of: a microphone, an inertial measurement unit, an outward-facing imaging system, or an inward-facing imaging system, and wherein the multimodal inputs comprise direct inputs comprising at least one of: a head pose, an eye gaze, a hand gesture, or a speech input.

14 . The system of claim 12 , wherein the multimodal inputs comprise at least one of: an input from a user input device or an indirect input, where the indirect input comprises a location of the user or a location of an object in the environment of the user.

15 . The system of claim 12 , wherein the subject comprises a target virtual object that the user intends to interact, wherein the command comprises an action that the wearable device performs on the target virtual object, and wherein the parameter comprises a characteristic of the action, and wherein the parameter comprises a placement, an orientation, a destination position of the target virtual object.

16 . The system of claim 12 , wherein the hardware processor is further configured to: detect an initiation condition associated with the user interaction, wherein the initiation condition comprises an invocation phrase or a hand gesture, and wherein the initiation condition comprises a combination of two input modes, wherein the two input modes indicate the user interaction.

17 . The system of claim 12 , wherein the hardware processor is further programmed to: receive environment data from an environmental sensor of the wearable device; and automatically setting an authentication level associated with an input mode based on the environment data.

18 . The system of claim 12 , wherein the user interaction comprises at least one of: selecting or moving the subject, conducting a telepresence session, modifying the subject, or sharing the subject with another wearable device.

19 . The system of claim 12 , wherein the multimodal inputs comprise head pose and speech input, and wherein the multimodal inputs further comprise a gesture or eye gaze.

Assignments (3)

Continuity (6)

References Cited (400)