Cross reality system with map processing using multi-resolution frame descriptors
A distributed, cross reality system efficiently and accurately compares location information that includes image frames. Each of the frames may be represented as a numeric descriptor that enables identification of frames with similar content. The resolution of the descriptors may vary for different computing devices in the distributed system based on degree of ambiguity in image comparisons and/or computing resources for the device. A descriptor computed for a cloud-based component operating on maps of large areas that can result in ambiguous identification of multiple image frames may use high resolution descriptors. High resolution descriptors reduce computationally intensive disambiguation processing. A portable device, which is more likely to operate on smaller maps and less likely to have the computational resources to compute a high resolution descriptor, may use a lower resolution descriptor.
1 . A system for providing shared location-based content, the system comprising:
a portable device, comprising:
at least one processor; and
at least one computer-readable medium comprising instructions that, when executed, cause the at least one processor to:
render virtual content within a representation of a 3D environment;
compute one or more feature descriptors each representing a respective feature of an image of the 3D environment;
send the one or more feature descriptors to a networked resource;
calculate a first frame descriptor from the one or more feature descriptors, where the first frame descriptor has a first resolution less than a second resolution of a second frame descriptor calculated for the image at the networked resource; and
localize the portable device by using the first frame descriptor, wherein the localizing is with respect to at least one of (i) a tracking map generated by the portable device, or (ii) at least a portion of a stored map identified by the network resource based on the second frame descriptor.
2 . The system of claim 1 , further comprising the networked resource, wherein the networked resource comprises at least one processor and at least one computer-readable medium comprising:
a plurality of stored maps of the 3 D environment, wherein at least one stored map of the plurality of stored maps is associated with at least one frame descriptor; and
computer executable instructions that, when executed by the at least one processor, cause the at least one processor to:
receive the one or more feature descriptors from the portable device;
calculate the second frame descriptor, the second frame descriptor being representative of the received one or more feature descriptors;
compare the second frame descriptor with the at least one frame descriptor associated with the at least one stored map of the 3 D environment; and
based on comparing the second frame descriptor with the at least one frame descriptor associated with the at least one stored map, localize the portable device to a shared coordinate system.
3 . The system of claim 2 , wherein the computer executable instructions of the networked resource, when executed by the at least one processor of the networked resource, cause the at least one processor of the networked resource to calculate the second frame descriptor using a neural network.
4 . The system of claim 2 , wherein the computer executable instructions of the networked resource, when executed by the at least one processor of the networked resource, further cause the at least one processor of the networked resource to:
receive the tracking map from the portable device;
merge the tracking map with the at least one stored map, to produce a merged map comprising location information based on location information of the at least one stored map and the tracking map; and
store the merged map in the at least one computer-readable medium of the networked resource.
5 . The system of claim 2 , wherein the second frame descriptor has a resolution greater than 512 bits.
6 . The system of claim 1 , wherein the instructions of the at least one computer-readable medium of the portable device, when executed by the at least one processor of the portable device, cause the at least one processor of the portable device to localize the portable device based on comparing the first frame descriptor to frame descriptors of image frames local to the portable device.
7 . The system of claim 1 , wherein the instructions of the at least one computer-readable medium of the portable device, when executed by the at least one processor of the portable device, further cause the at least one processor of the portable device to:
take the image of the 3D environment with at least one camera; and
identify a plurality of groups of pixels as representatives of features of the image,
wherein the one or more feature descriptors are computed as representing the plurality of groups of pixels.
8 . A system for providing shared location-based content to at least one portable device capable of rendering virtual content in a 3D environment, the system comprising:
one or more processors; and
at least one computer-readable medium comprising computer executable instructions that, when executed by the one or more processors, cause the one or more processors to:
receive, from a portable electronic device, one or more feature descriptors each representing a respective feature of an image of the 3D environment captured by the portable electronic device;
calculate a second frame descriptor for the image based on the received one or more feature descriptors; and
localize the portable electronic device to a shared coordinate system based on the second frame descriptor,
wherein the second frame descriptor has a second resolution greater than a first resolution of a first frame descriptor calculated at the at least one portable device.
9 . The system of claim 8 , wherein:
the at least one computer-readable medium further comprises a plurality of stored maps of the 3D environment; and
at least one map of the plurality of stored maps of the 3D environment is associated with at least one frame descriptor with a resolution greater than 512 bits.
10 . The system of claim 9 , wherein the computer executable instructions, when executed by the one or more processors, cause the one or more processors to localize the portable electronic device based on comparing compare the second frame descriptor for the image with the at least one frame descriptor associated with the at least one map of the plurality of stored maps of the 3D environment.
11 . The system of claim 10 , wherein the computer executable instructions, when executed by the one or more processors, cause the one or more processors to localize the portable electronic device by selecting one or more maps from the plurality of stored maps associated with the shared coordinate system, wherein the selecting is based on comparing the second frame descriptor for the image with the at least one frame descriptor associated with the at least one map of the plurality of stored maps.
12 . The system of claim 11 , wherein the computer executable instructions, when executed by the one or more processors, further cause the one or more processors to send, to the portable electronic device, the selected one or more maps.
13 . The system of claim 10 , wherein the computer executable instructions, when executed by the one or more processors, cause the one or more processors to localize the portable electronic device by determining whether a location of the portable electronic device corresponds to a stored map from the plurality of stored maps of the 3 D environment, wherein the determining is based on comparing the second frame descriptor for the image with the at least one frame descriptor associated with the at least one map of the plurality of stored maps.
14 . The system of claim 13 , wherein the computer executable instructions, when executed by the one or more processors, further cause the one or more processors to:
receive a tracking map from the portable electronic device;
merge the tracking map with the stored map, to produce a merged map comprising location information based on location information of the stored map and the tracking map; and
store the merged map in the at least one computer-readable medium.
15 . The system of claim 8 , wherein the portable electronic device is selected from a group comprising:
a wearable device comprising a head mounted display comprising a plurality of cameras mounted thereon; and
a portable computing device comprising a camera and a display and configured with computer-executable instructions for rendering virtual content on the display in connection with an image acquired by the camera.
16 . The system of claim 8 , wherein the computer executable instructions further comprise instructions for implementing a neural network to calculate the second frame descriptor for the image.
17 . At least one non-transitory computer-readable medium comprising computer executable instructions that, when executed by one or more processors, cause the one or more processors to:
receive, from a portable device, one or more feature descriptors each representing a respective feature of an image captured by the portable device; and
calculate a second frame descriptor for the image based on the received one or more feature descriptors; and
localize the portable device to a shared coordinate system based on the second frame descriptor,
wherein the second frame descriptor has a second resolution greater than a first resolution of a first frame descriptor calculated at the portable device.
18 . The at least one non-transitory computer-readable medium of claim 17 , further comprising a plurality of stored maps of 3 D environment, wherein:
the portable device is capable of rendering virtual content in the 3D environment; and
at least one map of the plurality of stored maps of the 3D environment is associated with at least one frame descriptor with a resolution greater than 512 bits.
19 . The at least one non-transitory computer-readable medium of claim 18 , wherein the computer executable instructions, when executed by the one or more processors, cause the one or more processors to localize the portable device based on comparing the second frame descriptor for the image with the at least one frame descriptor associated with the at least one map of the plurality of stored maps of the 3D environment.
20 . The at least one non-transitory computer-readable medium of claim 19 , wherein the computer executable instructions, when executed by the one or more processors, cause the one or more processors to, localize the portable device by selecting one or more maps from the plurality of stored maps associated with the shared coordinate system, wherein the selecting is based on comparing the second frame descriptor for the image with the at least one frame descriptor associated with the at least one map of the plurality of stored maps.