IP Library Granted Patent US 12,159,422
Granted Patent B2
US 12,159,422 · App. 17/944,037 · Granted Dec 3, 2024

Virtual and augmented reality systems and methods

Inventors: Jose Felix Rodriguez (Hialeah, FL); Ricardo Martinez Perez (Plantation, FL)
Assignee: Magic Leap, Inc.
G06T7/50G02B27/017G06T7/579G06T19/00G09G3/003G09G5/00H04N13/344H04N13/395H04N13/398G02B2027/0134G02B2027/0185G02B2027/0187G06F3/14G09G3/2022G09G2310/0235G09G2310/08G09G2320/0261G09G2340/02G09G2340/0407G09G2340/0464G09G2370/04
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,159,422
App. No.
17/944,037
Granted
Dec 3, 2024
Kind
B2
Abstract

A virtual or augmented reality display system can include a first sensor to provide measurements of a user's head pose over time and a processor to estimate the user's head pose based on at least one head pose measurement and based on at least one calculated predicted head pose. The processor can combine the head pose measurement and the predicted head pose using one or more gain factors. The one or more gain factors may vary based upon the user's head pose position within a physiological range of movement.

Claims (48)

1. A virtual or augmented reality display system comprising:

a first sensor configured to provide measurements of a head pose of a head of a user of the display system; and

a processor communicatively coupled to the first sensor; the processor configured to:

receive a plurality of head pose measurements generated by the first sensor, wherein the plurality of head pose measurements are of the head pose at different times during a time period;

calculate a predicted head pose of the user based on the plurality of head pose measurements; and

determine at least one estimated head pose of the user at at least one particular time based at least partly on at least one of the plurality of head pose measurements and the predicted head pose,

wherein determining the estimated head pose includes combining at least one of the head pose measurements and the predicted head pose using one or more gain factors that indicate a relative emphasis of the at least one head pose measurement and the predicted head pose in the combining, wherein a physiological range of possible head poses is mapped onto a three-dimensional surface for which each possible head pose is a normal vector that is perpendicular to the three-dimensional surface, with a neutral normal vector corresponding to a neutral head pose in which the user is looking directly forward, wherein the three-dimensional surface is divided into a plurality of non-overlapping regions that are each associated with a respective one or more different gain factors,

wherein the one or more gain factors associated with each of the regions varies based on a distance of the respective region from a region that includes the neutral normal vector, and

wherein the one or more gain factors used in the combining are determined as the one or more gain factors that are associated with the particular region that includes the at least one head pose measurement, wherein the one or more gain factors used in the combining are based on the distance of the particular region from the region that includes the neutral normal vector,

wherein determining the at least one estimated head pose further comprises:

determine one or more first gain factors based on the at least one head pose measurement indicating that the head pose is in a first region of the three-dimensional surface at a first time;

determine a first estimated head pose based on combining a first head pose measurement and a first predicted head pose using the one or more first gain factors; determine one or more second gain factors based on the at least one head pose measurement indicating that the head pose is in a second region of the three-dimensional surface at a second time different than the first time, the second region being different than the first region; and

determine a second estimated head pose based on combining a second head pose measurement and a second predicted head pose using the one or more second gain factors,

wherein the one or more first gain factors provide for a relative emphasis of the first predicted head pose over the first head pose measurement in the combining of the first head pose measurement and the first predicted head pose to determine the first estimated head pose,

wherein the one or more second gain factors provide for a relative emphasis of the second head pose measurement over the second predicted head pose in the combining of the second head pose measurement and the second predicted head pose to determine the second estimated head pose, and

wherein the first region is closer to the neutral normal vector than the second region.

2. The system of claim 1 , wherein the first sensor is configured to be head-mounted.

3. The system of claim 2 , wherein the first sensor comprises an inertial measurement unit.

4. The system of claim 1 , wherein the one or more gain factors emphasize the predicted head pose over the head pose measurement when the user's head pose is in a central portion of the physiological range of movement.

5. The system of claim 1 , wherein the one or more gain factors emphasize the predicted head pose over the head pose measurement when the user's head pose is nearer the middle of the physiological range of movement than a limit of the user's physiological range of movement.

6. The system of claim 1 , wherein the one or more gain factors emphasize the head pose measurement over the predicted head pose when the user's head pose approaches a limit of the physiological range of movement.

7. The system of claim 1 , wherein the one or more gain factors emphasize the head pose measurement over the predicted head pose when the user's head pose is nearer a limit of the physiological range of movement than the middle of the physiological range of movement.

8. The system of claim 1 , wherein the first sensor is configured to be head-mounted and further comprising a second sensor configured to be body-mounted, wherein the at least one head pose measurement is determined based on measurements from both the first sensor and the second sensor.

9. The system of claim 8 , wherein the head pose measurement is determined based on a difference between measurements from the first sensor and the second sensor.

10. A method of estimating head pose in a virtual or augmented reality display system, the method performed by at least one processor and comprising:

receiving, from a first sensor, a plurality of head poses measurements of a head pose of a head of a user of the display system, wherein the plurality of head pose measurements are of the head pose at different times during a time period;

calculating a predicted head pose of the user based on the plurality of head pose measurements; and

determining at least one estimated head pose of the user at at least one particular time based at least partly on at least one of the plurality of head pose measurements and the predicted head pose,

wherein determining the estimated head pose comprises combining at least one of the head pose measurements and the predicted head pose using one or more gain factors that indicate a relative emphasis of the at least one head pose measurement and the predicted head pose in the combining,

wherein a physiological range of possible head poses is mapped onto a three-dimensional surface for which each possible head pose is a normal vector that is perpendicular to the three-dimensional surface, with a neutral normal vector corresponding to a neutral head pose in which the user is looking directly forward, wherein the three-dimensional surface is divided into a plurality of non-overlapping regions that are each associated with a respective one or more different gain factors,

wherein the one or more gain factors associated with each of the regions varies based on a distance of the respective region from a region that includes the neutral normal vector, and wherein the one or more gain factors used in the combining are determined as the one or more gain factors that are associated with the particular region that includes the at least one head pose measurement,

wherein the one or more gain factors used in the combining are based on the distance of the particular region from the region that includes the neutral normal vector,

wherein determining the at least one estimated head pose further comprises:

determining one or more first gain factors based on the at least one head pose measurement indicating that the head pose is in a first region of the three-dimensional surface at a first time;

determining a first estimated head pose based on combining a first head pose measurement and a first predicted head pose using the one or more first gain factors;

determining one or more second gain factors based on the at least one head pose measurement indicating that the head pose is in a second region of the three-dimensional surface at a second time different than the first time, the second region being different than the first region; and

determining a second estimated head pose based on combining a second head pose measurement and a second predicted head pose using the one or more second gain factors,

wherein the one or more first gain factors provide for a relative emphasis of the first predicted head pose over the first head pose measurement in the combining of the first head pose measurement and the first predicted head pose to determine the first estimated head pose,

wherein the one or more second gain factors provide for a relative emphasis of the second head pose measurement over the second predicted head pose in the combining of the second head pose measurement and the second predicted head pose to determine the second estimated head pose, and

wherein the first region is closer to the neutral normal vector than the second region.

11. The method of claim 10 , where the first sensor is configured to be head-mounted, the method further comprising:

receiving body orientation measurements from a second sensor configured to be body-mounted; and

estimating the user's head pose based on the at least one head pose measurement and based on the at least one calculated predicted head pose,

wherein the at least one head pose measurement is determined based on measurements from both the first sensor and the second sensor.

12. The method of claim 10 , wherein the one or more gain factors emphasize the predicted head pose over the head pose measurement when the user's head pose is in a central portion of the physiological range of movement.

13. The method of claim 10 , wherein the one or more gain factors emphasize the predicted head pose over the head pose measurement when the user's head pose is nearer the middle of the physiological range of movement than a limit of the user's physiological range of movement.

14. The method of claim 10 , wherein the one or more gain factors emphasize the head pose measurement over the predicted head pose when the user's head pose approaches a limit of the physiological range of movement.

15. The method of claim 10 , wherein the one or more gain factors emphasize the head pose measurement over the predicted head pose when the user's head pose is nearer a limit of the physiological range of movement than the middle of the physiological range of movement.

Assignments (3)
SECURITY INTEREST Recorded Oct 29, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073438/0463 →
SECURITY INTEREST Recorded Feb 7, 2023
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 062681/0065 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 2, 2022
From: RODRIGUEZ, JOSE FELIX; PEREZ, RICARDO MARTINEZ
To: MAGIC LEAP, INC.
Reel/Frame 061962/0431 →
Continuity (3)
Continuation 15239710 · Aug 17, 2016
Provisional Application 62206765 · Aug 18, 2015
Related Publication 20230005167A1 · Jan 5, 2023
Cited By (1)
US 12,462,410