Cross reality system supporting multiple device types
A cross reality system enables any of multiple types of devices to efficiently and accurately access previously stored maps and render virtual content specified in relation to those maps. The cross reality system may include a cloud-based localization service that responds to requests from devices to localize with respect to a stored map. Devices of any type, with native hardware and software configured for augmented reality operations may be configured to work with the cross reality system by incorporating components that interface between the native AR framework of the device and the cloud-based localization service. These components may present position information about the device in a format recognized by the localization service. Additionally, these components may filter or otherwise process perception data provided by the native AR framework to increase the accuracy of localization.
1. A method of operating a portable electronic device of a first type in a cross reality system in which devices of multiple types interact with a remote localization service via at least one network, the portable electronic device being configured to capture images, the method comprising acts of:
capturing a plurality of images with a sensor on the portable electronic device, wherein the plurality of images comprise pixels depicting a three-dimensional (3D) environment of the portable electronic device at a time of capture;
modifying the plurality of images to normalize values of the pixels in the plurality of images with respect to values of pixels in images captured with electronic devices of a second type;
selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment;
generating feature descriptors from the selected clusters of pixels; and
causing the feature descriptors to be transmitted to the remote localization service via the at least one network.
2. The method of claim 1 , wherein modifying the plurality of images to normalize values of the pixels in the plurality of images with respect to values of pixels in images captured with electronic devices of a second type comprises:
modifying a brightness value for the pixels of the plurality of images captured by the sensor;
applying a gamma correction to the plurality of images; and/or
applying a denoise operation to the plurality of images.
3. The method of claim 2 , wherein selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment further comprises applying a lens shading correction operation to the plurality of images.
4. The method of claim 1 , wherein:
selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment comprises selecting clusters of pixels in the subset of the plurality of modified images, wherein the subset is formed, at least in part by filtering out images of the plurality of modified images based on motion blur detected in the images.
5. The method of claim 1 , wherein:
the portable electronic device is a handheld device; and
the second type is a wearable device comprising a head mounted display comprising a plurality of cameras mounted thereon.
6. The method of claim 1 , wherein generating feature descriptors from the selected clusters of pixels comprises generating location information associated with the feature descriptors.
7. The method of claim 1 , further comprising an act of:
obtaining from the remote localization service a transform between a stored coordinate frame of stored spatial information about the 3D environment and a local coordinate frame.
8. A portable electronic device of a first type in a cross reality system in which devices of multiple types interact with a remote localization service via at least one network, the portable electronic device being configured to capture images, comprising:
a sensor configured to capture a plurality of images;
at least one processor configured to execute computer executable instructions, wherein the computer executable instructions comprise instructions for:
capturing the plurality of images with the sensor on the portable electronic device, wherein the plurality of images comprise pixels depicting a three-dimensional (3D) environment of the portable electronic device at a time of capture;
modifying the plurality of images to normalize values of the pixels in the plurality of images with respect to values of pixels in images captured with electronic devices of a second type;
selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment;
generating feature descriptors from the selected clusters of pixels; and
causing the feature descriptors to be transmitted to the remote localization service via the at least one network.
9. The portable electronic device of claim 8 , wherein modifying the plurality of images to normalize values of the pixels in the plurality of images with respect to values of pixels in images captured with electronic devices of a second type comprises:
modifying a brightness value for the pixels of the plurality of images captured by the sensor;
applying a gamma correction to the plurality of images; and/or
applying a denoise operation to the plurality of images.
10. The portable electronic device of claim 9 , wherein selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment further comprises applying a lens shading correction operation to the plurality of images.
11. The portable electronic device of claim 8 , wherein the portable electronic device is a smartphone and the sensor is a camera on the smartphone, and wherein:
selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment comprises selecting clusters of pixels in the subset of the plurality of modified images, wherein the subset is formed, at least in part by filtering out images of the plurality of modified images based on motion blur detected in the images.
12. The portable electronic device of claim 8 , wherein:
the portable electronic device is a handheld device; and
the second type is a wearable device comprising a head mounted display comprising a plurality of cameras mounted thereon.
13. The portable electronic device of claim 8 , wherein generating feature descriptors from the selected clusters of pixels comprises generating location information associated with the feature descriptors.
14. The portable electronic device of claim 8 , wherein the computer executable instructions further comprise instructions for:
obtaining from the remote localization service a transform between a stored coordinate frame of stored spatial information about the 3D environment and a local coordinate frame.
15. A non-transitory computer-readable medium storing computer executable instructions configured to, when executed by at least one processor, perform a method for operating a portable electronic device of a first type in a cross reality system in which devices of multiple types interact with a remote localization service via at least one network, the portable electronic device being configured to capture images, the method comprising acts of:
capturing a plurality of images with a sensor on the portable electronic device, wherein the plurality of images comprise pixels depicting a three-dimensional (3D) environment of the portable electronic device at a time of capture;
modifying the plurality of images to normalize values of the pixels in the plurality of images with respect to values of pixels in images captured with electronic devices of a second type;
selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment;
generating feature descriptors from the selected clusters of pixels; and
causing the feature descriptors to be transmitted to the remote localization service via the at least one network.
16. The non-transitory computer-readable medium of claim 15 , wherein modifying the plurality of images to normalize values of the pixels in the plurality of images with respect to values of pixels in images captured with electronic devices of a second type comprises:
modifying a brightness value for the pixels of the plurality of images captured by the sensor;
applying a gamma correction to the plurality of images; and/or
applying a denoise operation to the plurality of images.
17. The non-transitory computer-readable medium of claim 16 , wherein selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment further comprises applying a lens shading correction operation to the plurality of images.
18. The non-transitory computer-readable medium of claim 15 , wherein:
selecting clusters of pixels in at least a subset of the plurality of modified images as representative of features in the 3D environment comprises selecting clusters of pixels in the subset of the plurality of modified images, wherein the subset is formed, at least in part by filtering out images of the plurality of modified images based on motion blur detected in the images.
19. The non-transitory computer-readable medium of claim 15 , wherein generating feature descriptors from the selected clusters of pixels comprises generating location information associated with the feature descriptors.
20. The non-transitory computer-readable medium of claim 15 , the method further comprising an act of:
obtaining from the remote localization service a transform between a stored coordinate frame of stored spatial information about the 3D environment and a local coordinate frame.