IP Library Granted Patent US 12,175,054
Granted Patent B2
US 12,175,054 · App. 18/331,308 · Granted Dec 24, 2024

Virtual user input controls in a mixed reality environment

Inventors: Adrian Kaehler (Los Angeles, CA); John Adam Croston (Seattle, WA)
Assignee: MAGIC LEAP, INC.
G06F3/04815G06F3/017G06N3/02G06N20/00G06T19/006G06V10/464G06V20/20G06V40/107
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,175,054
App. No.
18/331,308
Granted
Dec 24, 2024
Kind
B2
Abstract

A wearable display system includes a mixed reality display for presenting a virtual image to a user, an outward-facing imaging system configured to image an environment of the user, and a hardware processor operably coupled to the mixed reality display and to the imaging system. The hardware processor is programmed to generate a virtual remote associated with a parent device, render the virtual remote and the virtual control element on the mixed reality display, determine when the user of the wearable system interacts with the virtual control element of the virtual remote, and perform certain functions in response to user interaction with a virtual control element of the virtual remote. These functions may include generation the virtual control element to move on the mixed reality display; and when movement of the virtual control element surpasses a threshold condition, generate a focus indicator for the virtual control element.

Claims (48)

1. A wearable system for providing a virtual remote control in a mixed reality environment, the wearable system comprising:

a mixed reality display for presenting a virtual image to a user;

an outward-facing imaging system configured to image an environment of the user; and

a hardware processor operably coupled to the mixed reality display and to the outward-facing imaging system, the hardware processor programmed to:

generate a virtual remote associated with a parent device, the virtual remote comprising a virtual control element;

render the virtual remote and the virtual control element on the mixed reality display, wherein the virtual control element comprises a virtual button with an active surface and being movable through an active volume of space;

determine when the user of the wearable system interacts with the virtual control element of the virtual remote;

in response to user interaction with the virtual control element:

generate the virtual control element to move on the mixed reality display;

when movement of the virtual control element surpasses a threshold condition, generate a focus indicator for the virtual control element, the threshold condition comprising at least one of a threshold position, a threshold duration, or a threshold velocity, wherein the hardware processor is programmed to generate the focus indicator when movement of the virtual button exceeds the threshold condition, as taken in a direction normal to the active surface; and

transmit an instruction to the parent device to perform a function associated with the virtual control element in conjunction with at least one of movement of the virtual control element and generation of the focus indicator.

2. The wearable system of claim 1 , wherein the hardware processor is further programmed to generate the focus indicator in response to depression of the virtual button by the user or release of the virtual button by the user.

3. The wearable system of claim 1 , wherein the virtual button further comprises a distal surface; and

wherein the hardware processor is further programmed to vary an appearance of the focus indicator in conjunction with movement of the active surface relative to the distal surface.

4. The wearable system of claim 1 , wherein the threshold condition comprises a threshold duration; and

wherein the hardware processor is further programmed to generate the focus indicator in response to depression of the virtual button for a time period exceeding the threshold duration.

5. The wearable system of claim 1 , wherein the hardware processor is programmed to further generate graphics visually connecting the virtual remote to a real-world view of the parent device, as seen by the user through the wearable system.

6. The wearable system of claim 1 , wherein the hardware processor is further configured to:

identify a physical remote associated with a parent device; and

generate the virtual remote based at least partly on specifications of the physical remote.

7. The wearable system of claim 6 , wherein the hardware processor is programmed to generate the virtual input element to visually represent a physical input element of the physical remote.

8. The wearable system of claim 6 , wherein the hardware processor is programmed to further generate graphics visually connecting the virtual remote to a real-world view of the physical remote, as seen by the user through the wearable system.

9. A method for interacting with a virtual remote control in a mixed reality environment, the method carried out by a wearable system including a mixed reality display operably coupled to a hardware processor, the method comprising:

generating a virtual remote associated with a parent device, the virtual remote comprising a virtual control element;

rendering the virtual remote and the virtual control element on the mixed reality display, wherein the virtual control element comprises a virtual button having an active surface and being movable through an active volume of space;

determining when the user of the wearable system interacts with the virtual control element of the virtual remote;

in response to user interaction with the virtual control element:

generating the virtual control element to move on the mixed reality display;

when movement of the virtual control element surpasses a threshold condition, generating a focus indicator for the virtual control element, the threshold condition comprising at least one of a threshold position, a threshold duration, or a threshold velocity, wherein the hardware processor is programmed to generate the focus indicator when movement of the virtual button exceeds the threshold condition, as taken in a direction normal to the active surface; and

transmitting an instruction to the parent device to perform a function associated with the virtual control element in conjunction with at least one of movement of the virtual control element and generation of the focus indicator.

10. The method of claim 9 , wherein the hardware processor is further programmed to generate the focus indicator in response to at least one of depression of the virtual button by the user and release of the virtual button by the user.

11. The method of claim 9 , wherein the virtual button further comprises a distal surface; and

wherein the hardware processor is further programmed to vary an appearance of the focus indicator in conjunction with movement of the active surface relative to the distal surface.

12. The method of claim 9 , wherein the threshold condition comprises a threshold duration; and

wherein the hardware processor is further programmed to generate the focus indicator in response to depression of the virtual button for a time period exceeding the threshold duration.

13. The method of claim 9 , wherein the hardware processor is programmed to further generate graphics visually connecting the virtual remote to a real-world view of the parent device, as seen by the user through the wearable system.

14. The method of claim 9 , wherein the hardware processor is further configured to:

identify a physical remote associated with a parent device; and

generate the virtual remote based at least partly on specifications of the physical remote, while generating the virtual input element to visually represent a physical input element of the physical remote.

15. The method of claim 14 , wherein the hardware processor is programmed to further generate graphics visually connecting the virtual remote to a real-world view of the physical remote, as seen by the user through the wearable system.

16. Non-transitory computer storage media storing instructions that, when executed by one or more hardware processors of a wearable system, cause the wearable system to:

generate a virtual remote comprising a virtual control element on a mixed reality display of the wearable system, the virtual remote associated with a parent device, wherein the virtual control element comprises a virtual button with an active surface and being movable through an active volume of space;

render the virtual remote and the virtual control element on the mixed reality display;

determine when the user of the wearable system interacts with the virtual control element of the virtual remote;

in response to user interaction with the virtual control element:

generate the virtual control element to move on the mixed reality display;

when movement of the virtual control element surpasses a threshold condition, generate a focus indicator for the virtual control element, the threshold condition comprising at least one of a threshold position, a threshold duration, or a threshold velocity, wherein the generation of the focus indicator occurs when movement of the virtual button exceeds the threshold condition, as taken in a direction normal to the active surface; and

transmit an instruction to the parent device to perform a function associated with the virtual control element in conjunction with at least one of movement of the virtual control element and generation of the focus indicator.

Assignments (3)
SECURITY INTEREST Recorded Oct 29, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073438/0463 →
SECURITY INTEREST Recorded Oct 15, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073109/0238 →
PROPRIETARY INFORMATION AND INVENTIONS AGREEMENT Recorded Jun 8, 2023
From: KAEHLER, ADRIAN; CROSTON, JOHN ADAM
To: MAGIC LEAP, INC.
Reel/Frame 063917/0523 →
Continuity (5)
Continuation 17482256 · Sep 22, 2021
Continuation 15829249 · Dec 1, 2017
Provisional Application 62430308 · Dec 5, 2016
Provisional Application 62430279 · Dec 5, 2016
Related Publication 20230315250A1 · Oct 5, 2023