Cross reality system with localization service
A cross reality system enables any of multiple devices to efficiently and accurately access previously stored maps and render virtual content specified in relation to those maps. The cross reality system may include a cloud-based localization service that responds to requests from devices to localize with respect to a stored map. The request may include one or more sets of feature descriptors extracted from an image of the physical world around the device. Those features may be posed relative to a coordinate frame used by the local device. The localization service may identify one or more stored maps with a matching set of features. Based on a transformation required to align the features from the device with the matching set of features, the localization service may compute and return to the device a transformation to relate its local coordinate frame to a coordinate frame of the stored map.
1 . An electronic device configured to operate within a cross reality system, the electronic device comprising:
one or more sensors configured to capture information about a three-dimensional (3D) environment, the information comprising a plurality of images of the 3D environment; and
at least one processor configured to execute computer executable instructions, wherein the computer executable instructions comprise instructions for:
generating a local coordinate frame for representing positions in the 3D environment;
extracting up to a threshold number of features from each of the plurality of images of the 3D environment;
appending pose information to the extracted features, wherein the pose information indicates a pose of the electronic device in the local coordinate frame;
determining whether a threshold for one or more triggers is met to request a localization, wherein the threshold is set dynamically based on the extracted features;
in response to determining that the threshold is met, sending, over a network to a remote localization service configured to receive localization requests from a plurality of devices, information about the appended extracted features and position information for the extracted features expressed in the local coordinate frame; and
receiving, from the remote localization service, at least one transform relating the local coordinate frame to a second coordinate frame.
2 . The electronic device of claim 1 , wherein the computer executable instructions further comprise instructions for:
rendering a virtual content having a location specified in the second coordinate frame on a display of the electronic device in a position computed based at least in part on a transform of the at least one transform.
3 . The electronic device of claim 1 , wherein:
the computer executable instructions further comprise instructions for generating descriptors for the extracted features; and
sending information about the appended extracted features comprises sending the descriptors for the appended extracted features.
4 . The electronic device of claim 3 , wherein the computer executable instructions further comprise instructions for:
storing the information about the appended extracted features and the position information for the extracted features in a buffer, wherein:
sending over the network comprises sending together contents of the buffer such that the information about the appended extracted features and the position information for the extracted features are sent together.
5 . The electronic device of claim 4 , wherein:
the buffer comprises an adjustable size; and
the computer executable instructions further comprise instructions increasing a size of the buffer in response to a failure indication received over the network from the remote localization service.
6 . The electronic device of claim 1 , wherein the computer executable instructions further comprise instructions for:
increasing the threshold number in response to a failure indication received over the network from the remote localization service.
7 . The electronic device of claim 1 , wherein the computer executable instructions further comprise instructions for:
sending over the network to the remote localization service a request to initiate a session with the remote localization service, wherein the request to initiate the session with the remote localization service comprises an identifier of the electronic device and calibration data for the electronic device.
8 . The electronic device of claim 1 , wherein:
the information about the appended extracted features and the position information for the extracted features sent over the network comprise a request for localization services; and
the one or more trigger conditions comprise distance moved by the electronic device since a last successful request for localization.
9 . The electronic device of claim 1 , wherein the threshold is set to a first value for a first 3D environment and to a second value for a second 3D environment, the first value being lower than the second when fewer features are extracted from images of the first 3D environment than from images of the second 3D environment.
10 . A non-transitory computer-readable medium storing computer executable instructions configured to, when executed by at least one processor, perform a method for operating an electronic device configured to operate within a cross reality system, the method comprising:
receiving information about a three-dimensional (3D) environment, the information comprising a plurality of images of the 3D environment;
generating a local coordinate frame for representing positions in the 3D environment;
extracting up to a threshold number of features from each of the plurality of images of the 3D environment;
appending pose information to the extracted features, wherein the pose information indicates a pose of the electronic device in the local coordinate frame;
determining whether a threshold for one or more triggers is met to request a localization, wherein the threshold is set dynamically based on the extracted features;
in response to determining that the threshold is met, sending, over a network to a remote localization service configured to receive localization requests from a plurality of devices, information about the appended extracted features and position information for the extracted features expressed in the local coordinate frame; and
receiving, from the remote localization service, at least one transform relating the local coordinate frame to a second coordinate frame.
11 . The non-transitory computer-readable medium of claim 10 , wherein:
the method comprises sending, to the remote localization service, a request for localization services based on one or more trigger conditions being met;
the information about the appended extracted features and the position information for the extracted features sent over the network comprise a request for localization services; and
the one or more trigger conditions comprise distance moved by the electronic device since a last successful request for localization.
12 . The non-transitory computer-readable medium of claim 10 , wherein the method comprises:
increasing the threshold number in response to a failure indication received over the network from the remote localization service.
13 . A method for operating an electronic device configured to operate within a cross reality system, the method comprising:
receiving information about a three-dimensional (3D) environment, the information comprising a plurality of images of the 3D environment;
generating a local map having a local coordinate frame for representing positions in the 3D environment;
extracting a plurality of features from the plurality of images of the 3D environment;
appending pose information to the plurality of features, wherein the pose information indicates a pose of the electronic device in the local coordinate frame;
determining whether a threshold for one or more triggers is met to request a localization, wherein the threshold is set dynamically based on the extracted features;
in response to determining that the threshold is met, sending, over a network to a remote localization service, information about the appended plurality of features and position information for the plurality of features expressed in the local coordinate frame, wherein the information about the appended plurality of features and the position information for the plurality of features sent over the network comprises a request for localization services based, at least in part, on the one or more trigger conditions being met; and
receiving, from the remote localization service, at least one transform relating the local coordinate frame to a second coordinate frame.
14 . The method of claim 13 , further comprising:
rendering a virtual content having a location specified in the second coordinate frame on a display of the electronic device in a position computed based at least in part on a transform of the at least one transform.
15 . The method of claim 13 , further comprising:
generating descriptors for the extracted features, wherein:
sending information about the appended extracted features comprises sending the descriptors for the appended extracted features.
16 . The method of claim 15 , further comprising:
storing the information about the appended extracted features and the position information for the extracted features in a buffer, wherein:
sending over the network comprises sending together contents of the buffer such that the information about the appended extracted features and the position information for the extracted features are sent together.
17 . The method of claim 16 , wherein:
the buffer comprises an adjustable size; and
the method further comprises increasing a size of the buffer in response to a failure indication received over the network from the remote localization service.
18 . The method of claim 13 , wherein:
extracting the plurality of features from the plurality of images of the 3D environment comprises extracting up to a threshold number of features from each of the plurality of images of the 3D environment; and
increasing the threshold number in response to a failure indication received over the network from the remote localization service.
19 . The method of claim 13 , further comprising:
sending over the network to the remote localization service a request to initiate a session with the remote localization service, wherein the request to initiate the session with the remote localization service comprises an identifier of the electronic device and calibration data for the electronic device.
20 . The method of claim 13 , wherein:
the one or more trigger conditions comprise distance moved by the electronic device since a last successful request for localization.