Granted Patent B2

US 12,223,591 · App. 18/599,083 · Granted Feb 11, 2025

Inventors: Yilun Cao (San Jose, CA); Mohan Babu Kandra (San Ramon, CA); David Geoffrey Molyneaux (San Jose, CA); Daniel Olshansky (Mountain View, CA); David Paul Pena (Los Altos, CA); Frank Thomas Steinbrücker (Mountain View, CA); Rafael Domingos Torres (Boca Raton, FL)

Assignee: Magic Leap, Inc.

G06T17/00G06F3/012G06T7/55G06T19/006G06T2207/10028G06T2207/20221G06T2219/024

Abstract

Claims (93)

1. An extended-reality or cross-reality system comprising a wearable system for rendering virtual content, the wearable system further comprising:

a head unit comprising a display system and a plurality of sensor units and located in a physical environment in a first instance of the wearable system in a first session at a first time point; and

a non-transitory computer readable storage medium storing thereupon a plurality of modules, the plurality of modules comprising:

a coordinate system module that comprises a persistent coordinate frame (PCF) system;

a plurality of transform modules that performs one or more transformations pertaining to a local coordinate frame and a camera coordinate frame; and

a processor executing a sequence of instructions, wherein execution of the sequence of instructions causes the processor to perform a set of acts, the set of acts comprising:

persisting, by the persistent coordinate frame system, a virtual content in the physical environment from a first perspective when viewing the virtual content through the wearable system in the first instance of the wearable system in the first session, wherein

the virtual content is spatially persisted in the physical environment to prevent the virtual content from appearing out of place from a second perspective when viewing the virtual content from or through the wearable system in a second instance of the wearable system in a second session at a second time point based at least in part upon a result of the one or more transformations.

2. The extended-reality or cross-reality system of claim 1 , the set of acts further comprising determining the persistent coordinate frame system at least by:

identifying one or more points from the physical environment, wherein the one or more points representing at least one feature recognized by the head unit of the wearable system;

determining at least one degree of freedom of six degrees of freedom of the persistent coordinate frame system based at least in part upon a location at which the persistent coordinate frame system is stored a map coordinate system of the coordinate system module or a canonical map; and

determining the persistent coordinate frame system based at least in part upon the at least one degree of freedom.

3. The extended-reality or cross-reality system of claim 2 , the set of acts further comprising:

identifying the canonical map that is used in determining the at least one degree of freedom of the persistent coordinate frame system, wherein the canonical map includes a canonical map point that corresponds to a feature of an object;

localizing the canonical map to the wearable system for the first session; and

creating a new map at least by stitching the canonical map to the persistent coordinate frame system.

4. The extended-reality or cross-reality system of claim 2 , the set of acts further comprising:

identifying an object; and

representing the object as at least one persistent coordinate frame, wherein

the at least one persistent coordinate frame comprises a local coordinate frame relative to at least one coordinate frame in the plurality of coordinate system module, and

when the object is represented as multiple persistent coordinate frames, the multiple persistent coordinate frames respectively correspond to a plurality of features of the object that is represented as the multiple persistent coordinate frames.

5. The extended-reality or cross-reality system of claim 2 , the set of acts further comprising:

determining a plurality of canonical maps, wherein each canonical map of the plurality of canonical maps respectively represents a corresponding object or a feature of the corresponding object; and

providing a floorplan of a plurality of objects in the physical environment at least by geographically disposing the plurality of canonical maps in a two-dimensional pattern.

6. The extended-reality or cross-reality system of claim 2 , the set of acts further comprising:

determining a plurality of canonical maps;

determining, by one or more sensor units in the head unit of the wearable system, a tracking map that include an image frame, a keyframe, or data derived from the image frame or the keyframe at least by removing redundant information and information with an accuracy or resolution below a threshold from the image frame, the keyframe, or the data derived from the image frame or the keyframe;

ranking the plurality of canonical maps based at least in part upon respective similarities between one or more first regions in the plurality of canonical maps and one or more second regions in the tracking map; and

merging or stitching the tracking map into at least a tile in at least one canonical map of the plurality of canonical maps so as to have a correspondence between a first portion of the tracking map and a second portion of the at least one canonical map.

7. The extended-reality or cross-reality system of claim 6 , merging or stitching the tracking map into at least a tile in the at least one canonical map in the set of acts comprising:

aligning a set of tracking map points with a set of canonical map features in the at least the tile in the at least one canonical map at least by applying a transformation between the set of canonical map features and set of tracking map points; and

determining one or more overlapping portions of the tracking map and at least the tile of the at least one canonical map.

8. The extended-reality or cross-reality system of claim 1 , the set of acts further comprising:

determining, by one or more sensor units in the head unit of the wearable system, a tracking map that include an image frame, a keyframe, or data derived from the image frame or the keyframe at least by removing redundant information and information with an accuracy or resolution below a threshold from the image frame, the keyframe, or the data derived from the image frame or the keyframe;

refining the tracking map into a refined tracking map at least by correcting a drift or a deviation of a tracking path that deviates from an actual tracking path across at least a plurality of sessions that includes the first session;

merging or stitching the tracking map with an environment map, wherein the environment, wherein the environment map stores location or orientation data of at least a portion of the physical environment and data pertaining to gaze directions when perceived with the wearable system, and the environment map comprises more details about the physical world than the tracking map; and

creating or updating, by collaboration among a plurality of wearable systems in multiple sessions in the physical environment, a passable world by using at least the environment map, wherein the passable world so created or updated is shared among the plurality of wearable systems in the multiple sessions including the first and the second sessions.

9. The extended-reality or cross-reality system of claim 1 , the set of acts further comprising attaching the virtual content to the persistent coordinate frame system, attaching the virtual content to the persistent coordinate frame system comprising:

identifying a keyframe from a plurality of keyframes based at least in part upon a set of features in the plurality of image frames captured by one or sensor units in the head unit of the wearable system;

associating a pose of a sensor unit of the one or more sensor units with the keyframe;

generating a persistent pose from the keyframe based at least in part upon the pose and metadata about the keyframe, wherein the persistent pose includes a coordinate location or direction that has one or more associated keyframes; and

reflecting the persistent pose as a persistent coordinate frame of the persistent coordinate frame system, wherein the persistent coordinate frame comprises one or more features.

10. The extended-reality or cross-reality system of claim 9 , the set of acts further comprising attaching the virtual content to the persistent coordinate frame system, attaching the virtual content to the persistent coordinate frame system comprising:

orienting the wearable system to the persistent pose at least by correlating the wearable system using at least one or more features associated with the persistent pose in the keyframe;

determining, by the wearable system, an orientation of the wearable system with respect to the persistent coordinate frame, wherein the persistent coordinate frame comprises a transformation;

determining, by the wearable system, a position of the wearable system with respect to the virtual content at least by correlating the position to the persistent coordinate frame based at least in part upon the transformation included in the persistent coordinate frame; and

aligning, by the wearable system, a separate image to the persistent pose based at least in part upon a result of matching one or more characteristics of the keyframe and the separate image, wherein the separate image is captured by at least one of the one or more sensor units of the head unit.

11. The extended-reality or cross-reality system of claim 1 , the wearable system further comprising:

the persistent coordinate frame system that comprises a first surface determination routine and a first frame determination routine that is operatively connected to the surface determination routine, wherein

the first surface determination routine is used to determine a location of a surface in the physical environment, and

the first frame determination routine is used to determine a coordinate frame based at least in part upon the location of the surface determined by the surface determination routine;

a camera frame system that comprises camera intrinsics pertaining to the head unit and determines the camera coordinate frame based at least in part upon a local coordinate frame or a world coordinate frame.

12. The extended-reality or cross-reality system of claim 1 , the wearable system further comprising:

a local data system frame system that comprises a data channel receiving image data and a second frame determination routine that is operatively connected to the local data system and determines the local coordinate frame based at least in part upon at least one object in the physical environment or a location in the physical environment, wherein

the one or more transformations transform the local coordinate frame into the camera coordinate, and

the local coordinate frame defines a facing direction of the virtual content and a persistent coordinate frame node at which the virtual content is placed.

13. A method for rendering virtual content, comprising:

determining a local coordinate frame and a camera coordinate frame for a wearable system in a physical environment;

performing one or more transformations between the local coordinate frame and the camera coordinate frame; and

persisting, by a persistent coordinate frame system of the wearable system, a virtual content in the physical environment from a first perspective when viewing the virtual content through the wearable system in a first instance of the wearable system in the first session at a first time point, wherein

the virtual content is spatially persisted in the physical environment to prevent the virtual content from appearing out of place from a second perspective when viewing the virtual content from or through the wearable system in a second instance of the wearable system in a second session at a second time point based at least in part upon a result of the one or more transformations.

14. The method of claim 13 , further comprising:

determining the persistent coordinate frame system at least by:

identifying one or more points from the physical environment, wherein the one or more points representing at least one feature recognized by the head unit of the wearable system;

determining at least one degree of freedom of six degrees of freedom of the persistent coordinate frame system based at least in part upon a location at which the persistent coordinate frame system is stored a map coordinate system of the coordinate system module or a canonical map; and

determining the persistent coordinate frame system based at least in part upon the at least one degree of freedom.

15. The method of claim 14 , further comprising:

identifying the canonical map that is used in determining the at least one degree of freedom of the persistent coordinate frame system, wherein the canonical map includes a canonical map point that corresponds to a feature of an object;

localizing the canonical map to the wearable system for the first session; and

creating a new map at least by stitching the canonical map to the persistent coordinate frame system.

16. The method of claim 14 , further comprising:

identifying an object; and

representing the object as at least one persistent coordinate frame, wherein

the at least one persistent coordinate frame comprises a local coordinate frame relative to at least one coordinate frame in the plurality of coordinate system module, and

when the object is represented as multiple persistent coordinate frames, the multiple persistent coordinate frames respectively correspond to a plurality of features of the object that is represented as the multiple persistent coordinate frames.

17. The method of claim 14 , further comprising:

determining a plurality of canonical maps wherein each canonical map of the plurality of canonical maps respectively represents a corresponding object or a feature of the corresponding object; and

providing a floorplan of a plurality of objects in the physical environment at least by geographically disposing the plurality of canonical maps in a two-dimensional pattern.

18. The method of claim 14 , further comprising:

determining a plurality of canonical maps;

determining, by one or more sensor units in the head unit of the wearable system, a tracking map that include an image frame, a keyframe, or data derived from the image frame or the keyframe at least by removing redundant information and information with an accuracy or resolution below a threshold from the image frame, the keyframe, or the data derived from the image frame or the keyframe;

ranking the plurality of canonical maps based at least in part upon respective similarities between one or more first regions in the plurality of canonical maps and one or more second regions in the tracking map; and

merging or stitching the tracking map into at least a tile in at least one canonical map of the plurality of canonical maps so as to have a correspondence between a first portion of the tracking map and a second portion of the at least one canonical map.

19. The method of claim 13 , further comprising:

determining, by one or more sensor units in the head unit of the wearable system, a tracking map that include an image frame, a keyframe, or data derived from the image frame or the keyframe at least by removing redundant information and information with an accuracy or resolution below a threshold from the image frame, the keyframe, or the data derived from the image frame or the keyframe;

refining the tracking map into a refined tracking map at least by correcting a drift or a deviation of a tracking path that deviates from an actual tracking path across at least a plurality of sessions that includes the first session;

merging or stitching the tracking map with an environment map, wherein the environment, wherein the environment map stores location or orientation data of at least a portion of the physical environment and data pertaining to gaze directions when perceived with the wearable system, and the environment map comprises more details about the physical world than the tracking map; and

creating or updating, by collaboration among a plurality of wearable systems in multiple sessions in the physical environment, a passable world by using at least the environment map, wherein the passable world so created or updated is shared among the plurality of wearable systems in the multiple sessions including the first and the second sessions.

20. The method of claim 13 , further comprising:

identifying a keyframe from a plurality of keyframes based at least in part upon a set of features in the plurality of image frames captured by one or sensor units in the head unit of the wearable system;

associating a pose of a sensor unit of the one or more sensor units with the keyframe;

generating a persistent pose from the keyframe based at least in part upon the pose and metadata about the keyframe, wherein the persistent pose includes a coordinate location or direction that has one or more associated keyframes; and

reflecting the persistent pose as a persistent coordinate frame of the persistent coordinate frame system, wherein the persistent coordinate frame comprises one or more features.

Assignments (3)

Continuity (5)

References Cited (57)