Granted Patent B2

US 12,322,041 · App. 18/056,381 · Granted Jun 3, 2025

Inventors: Moshe Bouhnik (Holon, IL); Ben Weisbih (Rehovot, IL); Miguel Andres Granados Velasquez (Thalwil, CH); Ali Shahrokni (San Jose, CA); Ashwin Swaminathan (Dublin, CA)

Assignee: Magic Leap, Inc.

G06T17/05G06T7/55G06T15/005G06T19/006

Abstract

Claims (110)

1. A portable electronic system, comprising:

a processor;

a sparse map that is constructed by a portable electronic system and stored in a non-transitory computer readable medium, wherein the sparse map further comprises data of a first data type that includes location data that represents one or more features in a physical environment;

memory storing thereupon a sequence of instructions which, when executed by the processor, causes the processor to perform a set of acts, the set of acts comprising:

receiving, at the portable electronic system, a set of one or more canonical maps, wherein a canonical map includes data of a second data type, different from the first data type, that represents at least one feature of the one or more features in the physical environment;

determining, at the portable electronic system, the sparse map based at least in part upon one or more anchors according to the physical environment;

localizing the sparse map to at least one canonical map in the set of one or more canonical maps using at least a three-dimensional (3D) point cloud data structure; and

creating a new canonical map at least by merging sparse map data of the sparse map into the at least one canonical map.

2. The portable electronic system of claim 1 , receiving the set of one or more canonical maps comprising:

identifying a universe of canonical maps having a plurality of canonical maps, wherein the three-dimensional (3D) point cloud data structure is different from both the sparse map and the plurality of canonical maps in the universe of canonical maps;

determining one or more filtering schemes from a plurality of filtering schemes based at least in part upon a characteristic of the universe of canonical maps; and

filtering the universe of canonical maps at least by applying the one or more filtering schemes to the universe of canonical maps.

3. The portable electronic system of claim 2 , receiving the set of one or more canonical maps further comprising:

identifying a local feature string from the sparse map;

identifying a global feature string from a reduced set of canonical maps comprising the at least one canonical map; and

determining whether the local feature string matches the global feature string.

4. The portable electronic system of claim 3 , wherein the global feature string is locally generated by the portable electronic system.

5. The portable electronic system of claim 2 , determining the one or more filtering schemes comprising:

upon a determination that a first filtering scheme is to be applied to the universe of canonical maps;

identifying system location data or information pertaining to one or more prior maps to which the portable electronic system has been localized;

determining one or more neighboring canonical maps based at least by filtering the universe of canonical maps with the system location data of the portable electronic system, wherein a neighboring canonical map includes neighboring canonical map location data that encompasses or is within a close proximity of the system location data; and

identifying the one or more neighboring canonical maps into a reduced universe of canonical maps.

6. The portable electronic system of claim 5 , determining the one or more filtering schemes comprising:

upon a determination that a second filtering scheme is to be applied to the universe of canonical maps or to the reduced universe of canonical maps;

identifying wireless fingerprinting data for the portable electronic system, wherein the wireless fingerprinting data is determined based at least in part upon at least one of a basic service set identifier (BSSID), a Wi-Fi signal, a global positioning system (GPS) signal, or a received signal strength indicator (RSSI) signal;

determining a region in the physical environment based at least in part upon the wireless fingerprinting data for the portable electronic system; and

filtering the universe of canonical maps or the reduced universe of canonical maps into a first subset of canonical maps based at least in part upon the region determined from the physical environment.

7. The portable electronic system of claim 6 , determining the one or more filtering schemes comprising:

upon a determination that a third filtering scheme is to be applied to the universe of canonical maps, the reduced universe of canonical maps, or the first subset of canonical maps;

determining one or more keyframes, one or more two-dimensional (2D) or three-dimensional (3D) features, or one or more global feature strings, wherein a 2D or 3D feature comprises a localized pose and a set of 2D-to-3D correspondences; and

determining a second subset of one or more canonical maps from the universe of canonical maps, the reduced universe of canonical maps, or the first subset of canonical maps based at least in part upon the one or more keyframes, the one or more two-dimensional (2D) or three-dimensional (3D) features, or the one or more global feature strings.

8. The portable electronic system of claim 1 , localizing the sparse map to the at least one canonical map comprising:

identifying the sparse map and a universe of canonical maps having a plurality of canonical maps, wherein

the data of the first data type includes the location data that represents the one or more features in the physical environment, and

the data of the second data type includes a function or a transform that operates on an object represented in the canonical map; and

splitting the sparse map into a plurality of connected components, wherein a connected component of the plurality of connected components includes one or more three-dimensional (3D) features that are connected and located within a predetermined distance.

9. The portable electronic system of claim 8 , localizing the sparse map to the at least one canonical map further comprising:

determining a threshold for the plurality of connected components;

determining one or more connected components from the plurality of connected components that satisfy a constraint pertaining to the threshold;

merging the one or more connected components into one or more canonical maps; and

merging the one or more canonical maps into a merged canonical map.

10. The portable electronic system of claim 9 , localizing the sparse map to the at least one canonical map further comprising:

determining one or more remaining connected components from the plurality of connected components, wherein a remaining connected component does not satisfy the constraint pertaining to the threshold; and

promoting the one or more remaining connected components into a separate canonical map.

11. The portable electronic system of claim 10 , localizing the sparse map to the at least one canonical map further comprising:

identifying one or more sparse map poses or one or more sparse map persistent coordinate frames (PCFs) pertaining to the sparse map, wherein

the sparse map is determined based at least in part upon the one or more anchors, and

an anchor of the one or more anchors includes a persistent position or a persistent orientation with reference to the one or more sparse map persistent coordinate frames;

identifying one or more canonical map poses or one or more canonical map persistent coordinate frames (PCFs) pertaining to the merged canonical map; and

performing at least one of a first merger or a second merger.

12. The portable electronic system of claim 11 , wherein

the first merger merges the one or more sparse map poses pertaining to the sparse map into the one or more canonical map poses pertaining to the merged canonical map; and

the second merger merges the one or more sparse map PCFs pertaining to the sparse map into the one or more canonical map PCFs pertaining to the merged canonical map.

13. The portable electronic system of claim 11 , localizing the sparse map to the at least one canonical map further comprising fusing a plurality of map points in the merged canonical map based at least in part upon the first merger or the second merger.

14. The portable electronic system of claim 13 , localizing the sparse map to the at least one canonical map further comprising pruning at least one three-dimensional (3D) feature from the merged canonical map or from the merged canonical map.

15. A method, comprising:

identifying, from a non-transitory computer readable medium, a sparse map that is constructed by a portable electronic system, wherein the sparse map further comprises data of a first data type that includes location data that represents one or more features in a physical environment;

receiving, at the portable electronic system, a set of one or more canonical maps, wherein a canonical map includes data of a second data type, different from the first data type, that represents at least one feature of the one or more features in the physical environment;

determining, at the portable electronic system, the sparse map based at least in part upon one or more anchors according to the physical environment;

localizing the sparse map to at least one canonical map in the set of one or more canonical maps using at least a three-dimensional (3D) point cloud data structure; and

creating a new canonical map at least by merging sparse map data of the sparse map into the at least one canonical map.

16. The method of claim 15 , receiving the set of one or more canonical maps comprising:

identifying a universe of canonical maps having a plurality of canonical maps, wherein the three-dimensional (3D) point cloud data structure is different from both the sparse map and the plurality of canonical maps in the universe of canonical maps;

determining one or more filtering schemes from a plurality of filtering schemes based at least in part upon a characteristic of the universe of canonical maps;

filtering the universe of canonical maps at least by applying the one or more filtering schemes to the universe of canonical maps;

identifying a local feature string from the sparse map;

identifying a global feature string from a reduced set of canonical maps comprising the at least one canonical map; and

determining whether the local feature string matches the global feature string, wherein the global feature string is locally generated by the portable electronic system.

17. The method of claim 16 , receiving the set of one or more canonical maps comprising:

upon a first determination that a first filtering scheme is to be applied to the universe of canonical maps;

identifying system location data or information pertaining to one or more prior maps to which the portable electronic system has been localized;

determining one or more neighboring canonical maps based at least by filtering the universe of canonical maps with the system location data of the portable electronic system, wherein a neighboring canonical map includes neighboring canonical map location data that encompasses or is within a close proximity of the system location data; and

identifying the one or more neighboring canonical maps into a reduced universe of canonical maps;

upon a determination that a second filtering scheme is to be applied to the universe of canonical maps or to the reduced universe of canonical maps,

identifying wireless fingerprinting data for the portable electronic system, wherein the wireless fingerprinting data is determined based at least in part upon at least one of a basic service set identifier (BSSID), a Wi-Fi signal, a global positioning system (GPS) signal, or a received signal strength indicator (RSSI) signal;

determining a region in the physical environment based at least in part upon the wireless fingerprinting data for the portable electronic system; and

filtering the universe of canonical maps or the reduced universe of canonical maps into a first subset of canonical maps based at least in part upon the region determined from the physical environment; or

upon a determination that a third filtering scheme is to be applied to the universe of canonical maps, the reduced universe of canonical maps, or the first subset of canonical maps,

determining one or more keyframes, one or more two-dimensional (2D) or three-dimensional (3D) features, or one or more global feature strings, wherein a 2D or 3D feature comprises a localized pose and a set of 2D-to-3D correspondences; and

determining a second subset of one or more canonical maps the universe of canonical maps, the reduced universe of canonical maps, or the first subset of canonical maps based at least in part upon the one or more keyframes, the one or more two-dimensional (2D) or three-dimensional (3D) features, or the one or more global feature strings.

18. A computer program product comprising a non-transitory computer readable storage medium having stored thereupon a sequence of instructions which, when executed by a processor, causes the processor to perform a set of acts for rendering virtual content for extended-reality, the set of acts comprising:

identifying, from a non-transitory computer readable medium, a sparse map that is constructed by a portable electronic system, wherein the sparse map further comprises data of a first data type that includes location data that represents one or more features in a physical environment;

receiving, at the portable electronic system, a set of one or more canonical maps, wherein a canonical map includes data of a second data type, different from the first data type, that represents at least one feature of the one or more features in the physical environment;

determining, at the portable electronic system, the sparse map based at least in part upon one or more anchors according to the physical environment;

localizing the sparse map to at least one canonical map in the set of one or more canonical maps using at least a three-dimensional (3D) point cloud data structure; and

creating a new canonical map at least by merging sparse map data of the sparse map into the at least one canonical map.

19. The computer program product of claim 18 , further comprising the sequence of instructions for receiving the set of one or more canonical maps which, when executed by the processor, further causes the processor to perform the set of acts, the set of acts further comprising:

identifying a universe of canonical maps having a plurality of canonical maps, wherein the three-dimensional (3D) point cloud data structure is different from both the sparse map and the plurality of canonical maps in the universe of canonical maps;

determining one or more filtering schemes from a plurality of filtering schemes based at least in part upon a characteristic of the universe of canonical maps;

filtering the universe of canonical maps at least by applying the one or more filtering schemes to the universe of canonical maps;

identifying a local feature string from the sparse map;

identifying a global feature string from a reduced set of canonical maps comprising the at least one canonical map; and

determining whether the local feature string matches the global feature string, wherein the global feature string is locally generated by the portable electronic system.

20. The computer program product of claim 18 , further comprising the sequence of instructions for localizing the sparse map to the at least one canonical map which, when executed by the processor, further causes the processor to perform the set of acts, the set of acts further comprising:

identifying the sparse map and a universe of canonical maps having a plurality of canonical maps;

splitting the sparse map into a plurality of connected components, wherein a connected component of the plurality of connected components includes one or more three-dimensional (3D) features that are connected and located within a predetermined distance;

determining a threshold for the plurality of connected components;

determining one or more connected components from the plurality of connected components that satisfy a constraint pertaining to the threshold;

merging the one or more connected components into one or more canonical maps;

merging the one or more canonical maps into a merged canonical map;

determining one or more remaining connected components from the plurality of connected components, wherein a remaining connected component does not satisfy the constraint pertaining to the threshold;

promoting the one or more remaining connected components into a separate canonical map;

identifying one or more sparse map poses or one or more sparse map persistent coordinate frames (PCFs) pertaining to the sparse map, wherein

the sparse map is determined based at least in part upon the one or more anchors, and

an anchor of the one or more anchors includes a persistent position or a persistent orientation with reference to the one or more sparse map persistent coordinate frames;

identifying one or more canonical map poses or one or more canonical map persistent coordinate frames (PCFs) pertaining to the merged canonical map;

performing at least one of a first merger or a second merger, wherein

the first merger merges the one or more sparse map poses pertaining to the sparse map into the one or more canonical map poses pertaining to the merged canonical map; and

the second merger merges the one or more sparse map PCFs pertaining to the sparse map into the one or more canonical map PCFs pertaining to the merged canonical map; and

fusing a plurality of map points in the merged canonical map based at least in part upon the first merger or the second merger.

Assignments (4)

Continuity (3)

References Cited (28)