Cross reality system for large scale environment reconstruction

1. A method, comprising:

performing a spatial persistence process that places a virtual object relative to a physical object for an extended-reality display device based at least in part upon a persistent coordinate frame (PCF);

determining whether a drift is detected for the virtual object relative to the physical object; and

upon or after detection of the drift or deviation, correcting the drift or deviation at least by:

updating a tracking map into an updated tracking map; and

updating the persistent coordinate frame (PCF) based at least in part upon the updated tracking map, wherein

the persistent coordinate frame (PCF) comprises six degrees of freedom relative to a map coordinate system.

2. The method of claim 1 , wherein the PCF comprises six degrees of freedom relative to the map coordinate system.

3. The method of claim 1 , wherein

the tracking map is stored in a repository having stored thereupon a plurality of tracking maps,

the plurality of tracking maps is constructed with at least one extended-reality display device and a portable electronic device, and

the portable electronic device comprises a smart phone.

4. The method of claim 3 , wherein

the tracking map comprises one or more image frames or one or more keyframes or data derived from the one or more image frames or the one or more keyframes.

5. The method of claim 4 , further comprising:

generating a new tracking map at least by stitching the tracking map with a different tracking map in the plurality of tracking maps.

6. The method of claim 1 , further comprising:

identifying an environment map, wherein the environment map comprises data or information pertaining to a physical world, and the data or information pertaining to a physical environment comprises a location, an orientation, a depth, or a gaze direction for an object in the physical environment as perceived by the extended-reality device; and

extending the environment map at least by stitching the tracking map with the environment map.

7. The method of claim 6 , stitching the tracking map with the different tracking map comprising:

determining whether the tracking map and the different tracking map overlap; and

in response to a determination that the tracking map does not overlap the different tracking map,

promoting the tracking map to a canonical map, wherein the canonical map is generated using at least the persistent coordinate frame (PCF) or pose data pertaining to the virtual object or the physical object; and

merging the canonical map with the different tracking map.

8. The method of claim 7 , further comprising:

selecting a ranked environment map from the plurality of environment maps; and

updating the ranked environment map at least by merging the ranked environment map with the separate tracking map.

9. The method of claim 1 , further comprising:

identifying a separate tracking map from a repository having stored therein a plurality of tracking maps;

identifying a plurality of environment maps for the physical world; and

ranking the plurality of environment maps based at least in part upon relevance between the plurality of environment maps and the separate tracking map.

10. The method of claim 1 , further comprising:

identifying one or more canonical maps, wherein

the one or more canonical maps are created by one or more portable electronic devices that have entered at least respective portions of the physical environment before the extended-reality display device; and

determining one or more persistent coordinate frames in a canonical map of the one or more canonical maps.

11. The method of claim 10 , further comprising:

determining a correct location of the extended-reality display device in the physical environment using at least the canonical map, wherein

the one or more persistent coordinate frames comprise at least a unique identifier that is shared among the one or more portable electronic devices and the extended-reality display device,

the one or more persistent coordinate frames comprise at least a transformation relative to a reference location, an orientation, or a gaze direction pertaining to the physical object or the virtual object.

12. The method of claim 1 , further comprising:

determining whether the first sparse map and the second sparse map overlap; and

when it is determined that the first sparse map and the second sparse map overlap,

identifying one or more overlapping portions between the first sparse map and the second sparse map; and

merging the second sparse map with the first updated sparse map based at least in part upon the one or more overlapping portion.

13. The method of claim 1 , performing the spatial persistence process comprising:

determining the persistent coordinate frame (PCF) from a plurality of persistent coordinate frames pertaining to the physical environment;

identifying a stored location corresponding to the persistent coordinate frame (PCF); and

transforming the stored location corresponding to the persistent coordinate frame (PCF) into a current location for the extended-reality display device.

14. The method of claim 1 , further comprising:

determining a location for the extended-reality display device; and

deriving a feature for the location based at least in part upon feature data captured by the extended-reality display device.

15. The method of claim 1 , further comprising:

transforming the feature into the persistent coordinate frame (PCF), wherein the persistent coordinate frame (PCF) comprises a local coordinate frame for the extended-reality display device, and the local coordinate frame allows the spatial persistence process; and

associating the persistent coordinate frame (PCF) with an identifier that is shared among a plurality of portable electronic devices.

16. The method of claim 1 , further comprising:

in response to detecting that a portable electronic device or a separate extended-reality display device enters, using a map as a floorplan for the physical world, a separate location in the physical environment in which the extended-reality display device is located, determining, for the portable electronic device or the extended-reality display device, that sharable world data pertaining to the separate location has been accessed by one or more other portable electronic devices or one or more extended-reality display devices;

determining one or more persistent coordinate frames from a plurality of persistent coordinate frames for the separate location, wherein the one or more persistent coordinate frames comprise the persistent coordinate frame that is determined by the extended-reality display device;

determining one or more additional maps for the separate location; and

merging the map with the one or more additional maps for the separate location.

17. The method of claim 1 , merging the map with the one or more additional maps comprising:

grouping the one or more additional maps;

determining whether the map overlaps an additional map of the one or more additional maps; and

upon a determination that the map and the additional map overlap,

identifying an overlapping portion between the map and the additional map; and

merging the map with the additional map based at least in part upon the overlapping portion.

18. An extended reality or cross reality (XR) system for rendering virtual content, comprising:

a processor;

a memory storing thereupon a sequence of instructions which, when executed by the processor, causes the processor to perform a set of acts, the set of acts comprising:

performing a spatial persistence process that places a virtual object relative to a physical object for an extended-reality display device based at least in part upon a persistent coordinate frame (PCF);

determining whether a drift is detected for the virtual object relative to the physical object; and

upon or after detection of the drift or deviation, correcting the drift or deviation at least by:

updating a tracking map into an updated tracking map; and

updating the persistent coordinate frame (PCF) based at least in part upon the updated tracking map, wherein

the persistent coordinate frame (PCF) comprises six degrees of freedom relative to a map coordinate system.

19. The XR system of claim 18 , wherein

identifying an environment map, wherein the environment map comprises data or information pertaining to a physical world, and the data or information pertaining to a physical environment comprises a location, an orientation, a depth, or a gaze direction for an object in the physical environment as perceived by the extended-reality device;

extending the environment map at least by stitching the tracking map with the environment map; and

generating a new tracking map at least by stitching the tracking map with a different tracking map in the plurality of tracking maps.

20. A computer program product comprising a non-transitory computer readable storage medium having stored thereupon a sequence of instructions which, when executed by a microprocessor, causes the microprocessor to perform a set of acts that comprises:

performing a spatial persistence process that places a virtual object relative to a physical object for an extended-reality display device based at least in part upon a persistent coordinate frame (PCF);

determining whether a drift is detected for the virtual object relative to the physical object; and

upon or after detection of the drift or deviation, correcting the drift or deviation at least by:

updating a tracking map into an updated tracking map; and

updating the persistent coordinate frame (PCF) based at least in part upon the updated tracking map, wherein

the persistent coordinate frame (PCF) comprises six degrees of freedom relative to a map coordinate system.

21. The computer program product of claim 20 , wherein the set of acts further comprises:

identifying an environment map, wherein the environment map comprises data or information pertaining to a physical world, and the data or information pertaining to a physical environment comprises a location, an orientation, a depth, or a gaze direction for an object in the physical environment as perceived by the extended-reality device;

extending the environment map at least by stitching the tracking map with the environment map;

generating a new tracking map at least by stitching the tracking map with a different tracking map in the plurality of tracking maps;

determining whether the tracking map and the different tracking map overlap; and

in response to a determination that the tracking map does not overlap the different tracking map

promoting the tracking map to a canonical map, wherein the canonical map is generated using at least the persistent coordinate frame (PCF) or pose data pertaining to the virtual object or the physical object; and

merging the canonical map with the different tracking map.