Granted Patent B2

US 12,254,141 · App. 18/597,716 · Granted Mar 18, 2025

Inventor: Sheng Wan (San Jose, CA)

Assignee: Magic Leap, Inc.

G06F3/0346G06F3/012G06F3/0304G06T7/70G09G5/38G09G2340/0464G09G2354/00

Abstract

Claims (47)

1. A user interaction method comprising:

generating a totem inertial measurement unit (IMU) signal with a totem IMU on a totem body due to movement of the totem body;

locating a head unit on a head of a user;

storing a world frame;

generating, with a processor, a first location of the totem in the world frame based on the totem IMU data;

determining, with the processor, a second location of the totem relative to the world frame;

resetting, with the processor, sensors of the totem IMU to match the second location;

receiving, with the processor, image data from a data source; and

displaying, with the processor, a virtual object using the image data to a user, a location of the virtual object being based on the first location of the totem.

2. The method of claim 1 , further comprising:

comparing, with the processor, the first location and the second location;

declaring, with the processor, a drift only if the first location is more than a predetermined distance from the second location; and

resetting, with the processor, sensors of the totem IMU to match the second location only if the drift is declared.

3. The method of claim 1 , wherein a location of the totem relative to the head unit and a location of the head unit relative to the world frame are determined to establish a second location of the totem relative to the world frame.

4. The method of claim 1 , further comprising:

providing, with the processor, a data stream to a display system that includes the virtual object positioned at the first location.

5. The method of claim 1 , wherein the totem IMU reduces jitter of the virtual object.

6. The method of claim 1 , wherein the totem IMU includes at least one of a gyroscope and an accelerometer.

7. The method of claim 1 , further comprising:

transmitting an electromagnetic (EM) signal from an EM transmitter on the totem body;

receiving the EM signal with an EM receiver on the head unit body for the head unit, the EM wave being indicative of a location of the totem;

executing, with the processor, a fusion routine to generate the first location, which is a fused location of the totem in the world frame based on a combination of the EM wave with the totem IMU data; and

executing, with the processor, an unfused location determination modeler that determines a location of the totem relative to the head unit and a location of the head unit relative to the world frame to establish the second location, which is an unfused location of the totem relative to the world frame,

wherein a location of the virtual object is based on the fused location of the totem.

8. The method of claim 7 , wherein the fusion routine executes at a first frequency and the unfused location determination modeler executes at a second frequency that is different from the first frequency.

9. The method of claim 7 , wherein the EM receiver detects six-degrees of freedom (“6dof”) movement of the EM transmitter relative to the EM receiver.

10. The method of claim 7 , wherein the predetermined distance is less than 100 mm.

11. The method of claim 7 , further comprising:

storing, with the processor, a rig frame, being a mathematical object located between eyepieces of the head unit to serve as the basis for defining where an object lies relative to the head unit;

taking, with the processor, a measurement to determine a rig frame pose, being the rig frame relative to the world frame;

deriving, with the processor, a receiver-to-world pose using known extrinsics as provided by factory-level calibration for the relationship between the EM receiver and the rig frame;

estimating, with the processor, an EM relationship between the EM receiver and EM transmitter to derive a transmitter-in-world pose; and

deriving, with the processor, a totem IMU-in-world pose as a non-fused pose using known extrinsics of the totem for the relationship between the EM transmitter and the totem IMU.

12. The method of claim 1 , further comprising:

capturing an image of the totem with a camera on the head unit body; and

determining, with the processor, the unfused location based on the image of the totem.

13. The method of claim 1 , wherein the virtual object is displayed using the display system that includes:

a transparent waveguide secured to the head unit body that permits light from the totem through to an eye of a user wearing the head unit body; and

a projector that converts that image data to light, the light from the projector entering into the waveguide at an entry pupil and leaving the waveguide at an exit pupil to the eye of the user.

14. The method of claim 1 , further comprising:

detecting, with the processor, movement of the head unit;

calculating, with the processor, a placement value; and

modifying a position of the virtual object within a view of the user based on the placement value.

15. The method of claim 14 , wherein a head unit IMU of the head unit detects movement of the head-mountable frame.

16. The method of claim 15 , wherein the head unit IMU includes at least one of a gyroscope and an accelerometer.

17. The method of claim 14 , further comprising:

detecting, with a head unit camera, movement of the head unit by taking images of objects within view of the head unit camera, the position of the virtual object being modified based on the pose position.

Assignments (4)

Continuity (7)

References Cited (400)