Cross reality system with localization service
A cross reality system enables any of multiple devices to efficiently and accurately access previously stored maps and render virtual content specified in relation to those maps. The cross reality system may include a cloud-based localization service that responds to requests from devices to localize with respect to a stored map. The request may include one or more sets of feature descriptors extracted from an image of the physical world around the device. Those features may be posed relative to a coordinate frame used by the local device. The localization service may identify one or more stored maps with a matching set of features. Based on a transformation required to align the features from the device with the matching set of features, the localization service may compute and return to the device a transformation to relate its local coordinate frame to a coordinate frame of the stored map.
1. An electronic device configured to operate within a cross reality system, the electronic device comprising:
one or more sensors configured to capture information about a three-dimensional (3D) environment, the captured information comprising a plurality of images of the 3D environment;
at least one processor configured to execute computer executable instructions, wherein the computer executable instructions comprise instructions for:
generating a local coordinate frame for representing positions in the 3D environment;
extracting a plurality of features from the plurality of images of the 3D environment;
sending, over a network to a remote localization service having a persisted map with a second coordinate frame, information about the plurality of features and position information for the plurality of features expressed in the local coordinate frame, wherein the information about the plurality of features and the position information for the plurality of features sent over the network comprises a request for localization services and information about a subset of the plurality of features and position information for the subset of the plurality of features;
receiving, from the remote localization service, at least one transform relating the local coordinate frame for representing positions in the 3D environment to the second coordinate frame for representing locations of virtual content; and
applying a transform of the at least one transform to a location of virtual content, specified in the second coordinate frame, to compute a position in the 3D environment, specified in the local coordinate frame, for rendering the virtual content.
2. The electronic device of claim 1 , wherein:
the electronic device comprises a display; and
the computer-executable instructions further comprise instructions for rendering the virtual content having the location specified in the second coordinate frame on the display in the position computed based at least in part on the transform of the at least one transform.
3. The electronic device of claim 1 , wherein:
the computer executable instructions further comprise instructions for generating descriptors for the plurality of features; and
sending information about the plurality of features comprises sending the descriptors for the plurality of features.
4. The electronic device of claim 3 , wherein:
the computer executable instructions further comprise instructions for storing the information about the plurality of features and the position information for the plurality of features in a buffer; and
sending over a network comprises sending together contents of the buffer such that the information about the plurality of features and the position information for the plurality of features are sent together.
5. The electronic device of claim 4 , wherein:
the buffer comprises an adjustable size;
the computer executable instructions further comprise instructions for increasing the size of the buffer in response to a failure indication received over the network from the remote localization service.
6. The electronic device of claim 1 , wherein:
extracting the plurality of features from the plurality of images comprises extracting up to a threshold number of features from each image; and
the computer executable instructions further comprise instructions for increasing the threshold number in response to a failure indication received over the network from the remote localization service.
7. The electronic device of claim 1 , wherein the computer executable instructions further comprise instructions for sending over the network to the remote localization service a request to initiate a session with the remote localization service.
8. The electronic device of claim 1 , wherein the sent information about the plurality of features and position information for the plurality of features is not in a map.
9. The electronic device of claim 1 , wherein:
the computer executable instructions further comprise instructions for sending the request for localization services based on one or more trigger conditions being met.
10. The electronic device of claim 9 , wherein the one or more trigger conditions comprise distance moved by the electronic device since a last successful request for localization.
11. A method for operating an electronic device configured to operate within a cross reality system, the method comprising:
receiving information about a three-dimensional (3D) environment, the information comprising a plurality of images of the 3D environment;
generating a local coordinate frame for representing positions in the 3D environment;
extracting a plurality of features from the plurality of images of the 3D environment;
sending, over a network to a remote localization service configured to receive localization requests from a plurality of devices, information about the plurality of features and position information for the plurality of features expressed in the local coordinate frame, wherein the information about the plurality of features and the position information for the plurality of features sent over the network comprises a request for localization services and information about a subset of the plurality of features and position information for the subset of the plurality of features;
receiving, from the remote localization service, at least one transform relating the local coordinate frame for representing positions in the 3D environment to the second coordinate frame for representing locations of virtual content; and
applying a transform of the at least one transform to a location of virtual content, specified in the second coordinate frame, to compute a position in the 3D environment, specified in the local coordinate frame, for rendering the virtual content.
12. The method of claim 11 , further comprising:
rendering the virtual content having the location specified in the second coordinate frame on a display of the electronic device in the position computed based at least in part on the transform of the at least one transform.
13. The method of claim 11 , wherein:
the method further comprises generating descriptors for the plurality of features; and
sending information about the plurality of features comprises sending the descriptors for the plurality of features.
14. The method of claim 13 , wherein:
the method further comprises storing the information about the plurality of features and the position information for the plurality of features in a buffer; and
sending over a network comprises sending together contents of the buffer such that the information about the plurality of features and the position information for the plurality of features are sent together.
15. The method of claim 14 , wherein:
the buffer comprises an adjustable size;
the method further comprises increasing the size of the buffer in response to a failure indication received over the network from the remote localization service.
16. The method of claim 15 , wherein:
extracting the plurality of features from the plurality of images comprises extracting up to a threshold number of features from each image; and
the method further comprises increasing the threshold number in response to a failure indication received over the network from the remote localization service.
17. The method of claim 11 , wherein the method further comprises sending over the network to the remote localization service a request to initiate a session with the remote localization service, wherein the request to initiate the session with the remote localization service comprises an identifier of the electronic device and calibration data for the electronic device.
18. The method of claim 11 , wherein:
the method further comprises sending the request for localization services based on one or more trigger conditions being met.
19. The method of claim 18 , wherein the one or more trigger conditions comprise distance moved by the electronic device since a last successful request for localization.
20. A non-transitory computer-readable medium storing computer executable instructions configured to, when executed by at least one processor, perform a method for operating an electronic device configured to operate within a cross reality system, the method comprising:
receiving information about a three-dimensional (3D) environment, the information comprising a plurality of images of the 3D environment;
generating a local map having a local coordinate frame for representing positions in the 3D environment;
extracting a plurality of features from the plurality of images of the 3D environment;
sending, over a network to a remote localization service, information about the plurality of features and position information for the plurality of features expressed in the local coordinate frame, wherein the information about the plurality of features and the position information for the plurality of features sent over the network comprises a request for localization services and information about a subset of the plurality of features and position information for the subset of the plurality of features;
receiving, from the remote localization service, at least one transform relating the local coordinate frame for representing positions in the 3D environment to the second coordinate frame for representing locations of virtual content; and
applying a transform of the at least one transform to a location of virtual content, specified in the second coordinate frame, to compute a position with respect to the local map for rendering the virtual content at the location.