Granted Patent B2

US 12,699,497 · App. 18/948,996 · Granted Aug 4, 2026

Inventors: Adrian Kaehler (Los Angeles, CA); John Adam Croston (Seattle, WA)

Assignee: MAGIC LEAP, INC.

G06F3/04815G06F3/017G06N3/02G06N20/00G06T19/006G06V10/464G06V20/20G06V40/107

Abstract

Claims (62)

1 . A wearable system for providing a virtual remote control in a mixed reality environment, the wearable system comprising:

a mixed reality display for presenting a virtual image to a user;

an outward-facing imaging system configured to image an environment of the user; and

a hardware processor operably coupled to the mixed reality display and to the outward-facing imaging system, the hardware processor programmed to:

generate a virtual remote associated with a parent device, the virtual remote comprising a plurality of virtual control elements;

render the virtual remote and the virtual control elements on the mixed reality display;

determine when the user of the wearable system interacts with one of the virtual control elements of the virtual remote;

in response to user interaction with the one of the virtual control elements:

generate the virtual control element that the user has interacted with to move on the mixed reality display;

when movement of the virtual control element surpasses a threshold condition, generate a focus indicator for that visually indicates the virtual control element that the user has interacted with, the threshold condition comprising at least one of a threshold position or a threshold velocity;

transmit an instruction to the parent device to perform a function associated with the virtual control element that the user has interacted with in conjunction with at least one of movement of the virtual control element or generation of the focus indicator;

determine, after interaction of the user with the virtual remote, a period of inaction with respect to further interaction of the user with the virtual remote; and

removing the rendering of the virtual remote when the period of inaction exceeds a predetermined threshold.

2 . The wearable system of claim 1 , wherein the plurality of virtual control elements comprise a virtual button movable through an active volume of space.

3 . The wearable system of claim 2 , wherein the hardware processor is programmed to generate the focus indicator in response to depression of the virtual button by the user or release of the virtual button by the user.

4 . A wearable system for providing a virtual remote control in a mixed reality environment, the wearable system comprising:

a mixed reality display for presenting a virtual image to a user;

an outward-facing imaging system configured to image an environment of the user; and

a hardware processor operably coupled to the mixed reality display and to the outward-facing imaging system, the hardware processor programmed to:

generate a virtual remote associated with a parent device, the virtual remote comprising a plurality of virtual control elements;

render the virtual remote and the virtual control elements on the mixed reality display;

determine when the user of the wearable system interacts with one of the virtual control elements of the virtual remote;

in response to user interaction with the one of the virtual control elements:

generate the virtual control element that the user has interacted with to move on the mixed reality display;

when movement of the virtual control element surpasses a threshold condition, generate a focus indicator for that visually indicates the virtual control element that the user has interacted with, the threshold condition comprising at least one of a threshold position, a threshold duration, or a threshold velocity; and

transmit an instruction to the parent device to perform a function associated with the virtual control element that the user has interacted with in conjunction with at least one of movement of the virtual control element or generation of the focus indicator,

wherein the plurality of virtual control elements comprise a virtual button movable through an active volume of space,

wherein the virtual button comprises an active surface; and

wherein the hardware processor is programmed to generate the focus indicator when movement of the virtual button exceeds the threshold condition, as taken in a direction normal to the active surface.

5 . The wearable system of claim 4 , wherein the virtual button further comprises a distal surface; and

wherein the hardware processor is programmed to vary an appearance of the focus indicator in conjunction with movement of the active surface relative to the distal surface.

6 . The wearable system of claim 4 , wherein the threshold condition comprises a threshold duration; and

wherein the hardware processor is programmed to generate the focus indicator in response to depression of the virtual button for a time period exceeding the threshold duration.

7 . The wearable system of claim 1 , wherein the hardware processor is programmed to further generate graphics visually connecting the virtual remote to a real-world view of the parent device, as seen by the user through the wearable system.

8 . The wearable system of claim 1 , wherein the hardware processor is further configured to:

identify a physical remote associated with a parent device; and

generate the virtual remote based at least partly on specifications of the physical remote.

9 . The wearable system of claim 8 , wherein the hardware processor is programmed to generate the virtual control elements to visually represent a physical input element of the physical remote.

10 . The wearable system of claim 8 , wherein the hardware processor is programmed to further generate graphics visually connecting the virtual remote to a real-world view of the physical remote, as seen by the user through the wearable system.

11 . The wearable system of claim 1 , wherein the hardware processor is further programmed to, when removing the rendering of the virtual remote, gradually decrease visibility of the virtual remote.

12 . The wearable system of claim 11 , wherein the hardware processor is further programmed to, upon removing the rendering of the virtual remote, maintain the visual indication of the virtual control element.

13 . A method for interacting with a virtual remote control in a mixed reality environment, the method carried out by a wearable system including a mixed reality display operably coupled to a hardware processor, the method comprising:

generating a virtual remote associated with a parent device, the virtual remote comprising a plurality of virtual control elements;

rendering the virtual remote and the virtual control elements on the mixed reality display;

determining when the user of the wearable system interacts with one of the virtual control elements of the virtual remote;

in response to user interaction with the one of the virtual control elements:

generating the virtual control element that a user has interacted with to move on the mixed reality display;

when movement of the virtual control element surpasses a threshold condition, generating a focus indicator that visually indicates the virtual control element that the user has interacted with, the threshold condition comprising at least one of a threshold position, a threshold duration, or a threshold velocity; and

transmitting an instruction to the parent device to perform a function associated with the virtual control element that the user has interacted with in conjunction with at least one of movement of the virtual control element and generation of the focus indicator,

wherein the plurality of virtual control elements comprises a virtual button movable through an active volume of space,

wherein the virtual button comprises an active surface; and

wherein the hardware processor is programmed to generate the focus indicator when movement of the virtual button exceeds the threshold condition, as taken in a direction normal to the active surface.

14 . The method of claim 13 , wherein the hardware processor is programmed to generate the focus indicator in response to at least one of depression of the virtual button by the user and release of the virtual button by the user.

15 . The method of claim 13 , wherein the virtual button further comprises a distal surface; and

wherein the hardware processor is programmed to vary an appearance of the focus indicator in conjunction with movement of the active surface relative to the distal surface.

16 . The method of claim 13 , wherein the threshold condition comprises a threshold duration; and

wherein the hardware processor is programmed to generate the focus indicator in response to depression of the virtual button for a time period exceeding the threshold duration.

17 . The method of claim 13 , wherein the hardware processor is programmed to further generate graphics visually connecting the virtual remote to a real-world view of the parent device, as seen by the user through the wearable system.

18 . The method of claim 13 , wherein the hardware processor is further configured to:

identify a physical remote associated with a parent device; and

generate the virtual remote based at least partly on specifications of the physical remote, while generating the virtual control elements to visually represent a physical input element of the physical remote.

19 . The method of claim 18 , wherein the hardware processor is programmed to further generate graphics visually connecting the virtual remote to a real-world view of the physical remote, as seen by the user through the wearable system.

Assignments (2)

Continuity (6)

References Cited (303)