Granted Patent B2

US 12,283,012 · App. 18/510,623 · Granted Apr 22, 2025

Inventors: Xuan Zhao (San Jose, CA); Christian Ivan Robert Moore (Cupertino, CA); Sen Lin (San Jose, CA); Ali Shahrokni (San Jose, CA); Ashwin Swaminathan (Dublin, CA)

Assignee: Magic Leap, Inc.

G06T19/006G06F9/547G06F16/29G06F16/907

Abstract

Claims (72)

1. A cloud service for a cross reality system, the service comprising:

computer storage media storing:

a database of maps;

a plurality of types of location metadata associated with maps of the database, the plurality of types of location metadata comprising wireless fingerprints and geo-location information;

one or more computing devices configured for network communication with a plurality of portable electronic devices, the one or more computing devices comprising non-transitory computer-readable media comprising computer-executable instructions that, when executed perform a method comprising:

receiving, from a portable electronic device of the plurality of portable electronic devices, a communication comprising position information for the device and location metadata;

selecting, for the communication, a type of location metadata of the plurality of types of location metadata;

selecting a set of maps from the database based on the received location metadata and the selected type of location metadata associated with maps of the database; and

comparing the received position information to maps of the set.

2. The cloud service of claim 1 , wherein the received position information comprises a device tacking map.

3. The cloud service of claim 2 , wherein:

comparing comprises identifying a matching map from the set; and

the method further comprises merging the tracking map with the matching map.

4. The cloud service of claim 3 , wherein merging the tracking map with the matching map comprises:

associating location metadata from the tracking map with the merged map in the database of maps; and

storing geo-location information derived from the tracking map as an index of a record of a geo-location database, wherein the record identifies at least a portion of the merged map.

5. The cloud service of claim 1 , wherein:

the comparing comprises attempting to localize the device to a map of the set; and

the method further comprises, based on successfully localizing the device to a location in a stored map, transmitting to the device geo-location information derived from location metadata stored in association with the location in the stored map.

6. The cloud service of claim 5 , wherein:

the method further comprises, based on successfully localizing the device to a location in a stored map, storing geo-location information derived from the received location metadata as location metadata associated with the location in the stored map.

7. The cloud service of claim 6 , wherein:

storing geo-location information as location metadata associated with the location in the stored map comprises:

hashing the geo-location information; and

updating a database indexed by geo-hash value with the hashed geo-location information.

8. The cloud service of claim 5 , wherein:

the method further comprises, based on unsuccessfully localizing the device, storing geo-location information derived from the received location metadata as location metadata associated with a location in a stored map against which the device previously successfully localized.

9. A system comprising a plurality of portable electronic devices operating in a physical environment and configured to operate within a cross reality system to provide a shared experience of virtual content to users of the plurality of portable electronic devices, wherein:

each of the portable electronic device comprises:

one or more sensors configured to capture information about an environment, the captured information comprising a plurality of images; and

at least one processor configured to execute computer-executable instructions, wherein the computer-executable instructions comprise instructions for:

extracting a plurality of features from images captured with the one or more sensors;

generating location metadata indicative of a physical location of the portable electronic device;

associating the location metadata with the plurality of features;

communicating over a network, to a localization service:

information about the plurality of features, and

the location metadata associated with the plurality of features;

receiving, from the localization service, at least one transform relating a local coordinate frame of the portable electronic device to a second coordinate frame; and

rendering virtual content using the at least one transform, wherein rendering the virtual content using the at least one transform causes the virtual content to be positioned and/or oriented in the local coordinate frame such that the users of the plurality of portable electronic devices share a same relative position and/or orientation of the virtual content with respect to the physical environment.

10. The portable electronic device of claim 9 , wherein the location metadata includes prior localization information.

11. The portable electronic device of claim 9 , wherein:

the portable electronic device comprises an operating system with a geo-location application programming interface (API); and

the computer-executable instructions further comprise instructions for obtaining the location metadata by calling the geo-location API.

12. The portable electronic device of claim 9 , wherein:

the computer-executable instructions further comprise instructions for receiving a description of virtual content in a location based on the second coordinate frame and rendering the virtual content based on the at least one transform.

13. The portable electronic device of claim 9 , wherein:

the portable electronic device comprises a buffer configured to store a plurality of data structures; and

associating the location metadata with the plurality of features comprises storing, in a data structure of the plurality of data structures, in the buffer, the location metadata, the information about the plurality of features, and the position information for the plurality of features.

14. The portable electronic device of claim 13 , wherein:

communicating over the network to the localization service comprises transmitting the plurality of data structures from the buffer as a localization request.

15. The portable electronic device of claim 9 , wherein:

the computer-executable instructions further comprise instructions for:

constructing a local device map from the plurality of images,

defining persistent locations in the local device map, and

associating geo-location information with the persistent locations; and

associating the location metadata with the plurality of features comprises accessing, from the local device map, geo-location information associated with a persistent location of the persistent locations proximate to the portable electronic device coincident in time with capture of the plurality of images from which the plurality of features were extracted.

16. The portable electronic device of claim 9 , wherein the computer-executable instructions comprise instructions for generating or receiving geolocation information on a current location of the portable electronic device in the 3D environment.

17. A cross reality system for providing a shared experience of virtual content to users of a plurality of portable electronic devices, the system comprising:

a computing device configured to communicate with a plurality of devices through a communication network, the computing device comprising:

at least one processor configured to:

receive, from the plurality of portable electronic devices:

information about features in images captured by the plurality of portable electronic devices; and

location metadata associated with the features;

determine, using the information about the features in images captured by the plurality of portable electronic devices and the location metadata associated with the features, at least one transform for each of the plurality of portable electronic devices to obtain a plurality of transforms;

transmit, through the communication network, the plurality of transforms to respective ones of the plurality of portable electronic devices, wherein the plurality of transforms, when used to render virtual content by the respective portable electronic devices, cause users of the plurality of portable electronic devices to experience a same relative location and/or orientation of the virtual content.

18. The cross reality system of claim 17 , wherein determining, using the information about the features in the images captured by the plurality of portable electronic devices and the location metadata associated with the features, at least one transform for each of the plurality of portable electronic devices to obtain the plurality of transforms comprises:

localizing each of the plurality of portable electronic devices to a map of a set of maps; and

determining at least one transform for each of the portable electronic devices using a respective map to which the portable electronic device was localized.

19. The cross reality system of claim 17 , wherein the location metadata associated with the features comprises geo-location information from the plurality of portable electronic devices.

20. The cross reality system of claim 19 , wherein determining, using the information about the features in images captured by the plurality of portable electronic devices and the location metadata associated with the features, at least one transform for each of the plurality of portable electronic devices to obtain a plurality of transforms comprises:

querying a geo-location database based on the location metadata to identify a map of a set of maps for each of the plurality of portable electronic devices; and

determining the at least one transform for each of the plurality portable electronic devices using a map identified for the portable electronic device.

Assignments (3)

Continuity (3)

References Cited (344)