Cross reality system with prioritization of geolocation information for localization
A cross reality system enables any of multiple devices to efficiently access previously stored maps. Both stored maps and tracking maps used by portable devices may have any of multiple types of location metadata associated with them. The location metadata may be used to select a set of candidate maps for operations, such as localization or map merge, that involve finding a match between a location defined by location information from a portable device and any of a number of previously stored maps. The types of location metadata may prioritized for use in selecting the subset. To aid in selection of candidate maps, a universe of stored maps may be indexed based on geo-location information. A cross reality platform may update that index as it interacts with devices that supply geo-location information in connection with location information and may propagate that geo-location information to devices that do not supply it.
1. A portable electronic device configured to operate within a cross reality system, the portable electronic device comprising:
one or more sensors configured to capture information about a three-dimensional (3D) environment, the captured information comprising a plurality of images; and
at least one processor configured to execute computer executable instructions, wherein the computer executable instructions comprise instructions for:
extracting a plurality of features from the plurality of images of the 3D environment associated with the portable electronic device;
generating location metadata indicative of a physical location of the portable electronic device when the one or more sensors captured the plurality of images;
associating the location metadata with the plurality of features;
communicating over a network, to a localization service:
information about the plurality of features,
position information for the plurality of features expressed in a local coordinate frame of the portable electronic device, and
the location metadata associated with the plurality of features;
receiving, from the localization service, at least one transform relating the local coordinate frame to a second coordinate frame; and
rendering virtual content with a position and/or orientation determined based on the received at least one transform.
2. The portable electronic device of claim 1 , wherein the location metadata includes prior localization information.
3. The portable electronic device of claim 1 , wherein:
the portable electronic device comprises an operating system with a geo-location application programming interface (API);
the computer executable instructions further comprise instructions for obtaining the location metadata by calling the geo-location API.
4. The portable electronic device of claim 1 , wherein:
the computer executable instructions further comprise instructions for receiving a description of virtual content in a location based on the second coordinate frame and rendering the virtual content based on the at least one transform.
5. The portable electronic device of claim 1 , wherein:
the portable electronic device comprises a buffer configured to store a plurality of data structures; and
associating the location metadata with the plurality of features comprises storing in a data structure of the plurality of data structures in the buffer the location metadata, the information about the plurality of features, and the position information for the plurality of features.
6. The portable electronic device of claim 5 , wherein:
communicating over the network to the localization service comprises transmitting the plurality of data structures from the buffer as a localization request.
7. The portable electronic device of claim 1 , wherein:
the computer executable instructions further comprise instructions for:
constructing a local device map from the plurality of images,
defining persistent locations in the local device map, and
associating geo-location information with the persistent locations; and
associating the location metadata with the plurality of features comprises accessing, from the local device map, geo-location information associated with a persistent location of the persistent locations proximate to the portable electronic device coincident in time with capture of the plurality of images from which the plurality of features were extracted.
8. The portable electronic device of claim 1 , wherein:
the computer executable instructions further comprise instructions for obtaining the location metadata by filtering, clustering, and/or normalizing location information.
9. The portable electronic device of claim 1 , wherein the computer executable instructions comprise generating or receiving geolocation information on a current location of the portable electronic device in the 3D environment.
10. A cross reality system that supports specification of a position of virtual content relative to a plurality of stored maps previously created to describe one or more three-dimensional (3D) environments, the plurality of stored maps being associated with features, the system comprising:
one or more computing devices configured for network communication with one or more portable electronic devices, the one or more computing devices comprising:
a communication component configured to receive, from a portable electronic device, information about a set of features in a 3D environment of the portable electronic device, position information for the set of features expressed in a first coordinate frame, and location metadata descriptive of one or more physical world locations;
a localization component, connected to the communication component, the localization component configured to:
select a set of stored maps from among the plurality of stored maps based on the received location metadata and based on location metadata associated with the plurality of stored maps;
identify a stored map from the set of stored maps based on matching the received set of features to features of the identified stored map, wherein the identified stored map comprises a second coordinate frame;
generate a transform between the first coordinate frame and the second coordinate frame based on a computed alignment between the received set of features and the matching set of features in the identified stored map; and
send the transform to the portable electronic device.
11. The system of claim 10 , wherein:
the localization component is configured to select a type of location metadata for the selecting the set of stored maps based on a priority associated with types of location metadata.
12. The system of claim 11 , wherein:
the priority associated with the types of location metadata prioritizes prior localization information over a WiFi fingerprint.
13. The system of claim 12 , wherein:
the priority associated with the types of location metadata prioritizes the WiFi fingerprint over geo-location information obtained from an operating system of the portable electronic device.
14. The system of claim 10 , wherein:
the one or more computing devices further comprise a geo-location database identifying the stored maps by geo-location.
15. The system of claim 14 , wherein the localization component is further configured to select the set of stored maps by querying the geo-location database based on the received location metadata.
16. The system of claim 15 , wherein the localization component is further configured to update the geo-location database based on the received location metadata and the identifying the stored map.
17. The system of claim 15 , wherein the localization component is further configured to send geo-location information to the portable electronic device based on location metadata associated with the identified stored map.
18. The system of claim 10 , wherein the localization component is further configured to update the location metadata associated with the stored maps based on the received location metadata and the identifying the stored map.
19. The system of claim 18 , wherein the localization component is further configured to:
identify a localization failure; and
update the location metadata associated with the stored maps responsive to a localization failure based on prior successful localization information for the portable electronic device.
20. The system of claim 10 , wherein:
the stored maps comprise a plurality of segments; and
selecting a set of stored maps comprises selecting tiles of the stored maps.