Location determination and mapping with 3D line junctions
A system and method for determining a location of a client device is described herein. In particular, a client device receives images captured by a camera at the client device. The client device identifies features in the images. The features may be line junctions, lines, curves, or any other features found in images. The client device retrieves a 3D map of the environment from the map database and compares the identified features to the 3D map of the environment, which includes map features such as map line junctions, map lines, map curves, and the like. The client device identifies a correspondence between the features identified from the images and the map features and determines a location of the client device in the real world based on the correspondence. The client device may display visual data representing a location in a virtual world corresponding to the location in the real world.
1. A method for locating a camera in an environment, the method comprising:
comparing a feature identified in an image captured by the camera to a 3D map of the environment;
identifying a correspondence between the feature and a 3D line junction in the 3D map by iteratively projecting endpoints from 3D to 2D using respective predicted poses to identify a predicted pose whose calculation error is below a threshold; and
determining a pose of the camera within the environment using the correspondence.
2. The method of claim 1 , wherein the camera is included in a client device, and wherein the method further comprises:
receiving a plurality of images captured by the camera of the client device; and
identifying features in the plurality of images, wherein the identified features are compared to the 3D map of the environment to identify the correspondence.
3. The method of claim 2 , wherein identifying the features in the plurality of images comprises identifying line junctions in the plurality of images, and wherein the method further comprises:
comparing the identified line junctions to the 3D map of the environment by determining endpoints within the 3D map and using the endpoints as vertices for 3D line junctions in the 3D map.
4. The method of claim 3 , wherein identifying the line junctions in the plurality of images comprises:
determining 2D line equations of lines in the plurality of images, wherein each 2D line equation describes a normal vector in 3D space;
determining 2D line junctions in each of the plurality of images, wherein each 2D line junction is comprised of two 2D line equations and an intersection point; and
triangulating the 2D line junctions into 3D space to obtain 3D line junctions.
5. The method of claim 4 , wherein triangulating the 2D line junctions comprises:
converting coordinates of the 2D line junctions from each of the plurality of images to one camera coordinate space; and
transforming the converted 2D line junctions to 3D line junctions.
6. The method of claim 1 , wherein the feature is a line junction identified in the image.
7. The method of claim 6 , wherein the identified line junction has seven degrees of freedom, wherein three degrees of freedom are for a junction point, and two degrees of freedom are for each of two direction vectors.
8. The method of claim 1 , further comprising:
creating a representation of the environment around the camera by projecting the image into a graph of vertices and edges;
wherein the feature is identified from the representation, the feature comprising overlapping edges in the representation.
9. The method of claim 1 , wherein the camera is included in a client device, and wherein the method further comprises:
receiving image data from an additional client device connected to the client device for a virtual game.
10. The method of claim 1 , wherein comparing the feature to the 3D map of the environment comprises:
applying pattern matching to the feature and the 3D map to determine whether the feature and the 3D map overlap or are adjacent.
11. The method of claim 1 , wherein the determined pose corresponds to a location in a virtual world.
12. The method of claim 11 , further comprising:
sending, to a client device including the camera, a view of the location in the virtual world corresponding to the determined pose.
13. A non-transitory computer-readable storage medium comprising instructions for locating a camera in an environment, the instructions, when executed by one or more processors of a client device, cause the client device to perform operations comprising:
comparing a feature identified in an image captured by the camera to a 3D map of the environment;
identifying a correspondence between the feature and a 3D line junction in the 3D map by iteratively projecting endpoints from 3D to 2D using respective predicted poses to identify a predicted pose whose calculation error is below a threshold; and
determining a pose of the camera within the environment using the correspondence.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the camera is included in the client device, and wherein the instructions further cause the client device to perform operations comprising:
receiving a plurality of images captured by the camera of the client device; and
identifying features in the plurality of images, wherein the identified features are compared to the 3D map of the environment to identify the correspondence.
15. The non-transitory computer-readable storage medium of claim 14 , wherein the instructions for identifying the features in the plurality of images comprise instructions for identifying line junctions in the plurality of images, and wherein the instructions further cause the client device to perform an operation comprising:
comparing the identified line junctions to the 3D map of the environment by determining endpoints within the 3D map and using the endpoints as vertices for 3D line junctions in the 3D map.
16. The non-transitory computer-readable storage medium of claim 13 , wherein the feature is a line junction identified in the image.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the identified line junction has seven degrees of freedom, wherein three degrees of freedom are for a junction point, and two degrees of freedom are for each of two direction vectors.
18. The non-transitory computer-readable storage medium of claim 13 , wherein the determined pose corresponds to a location in a virtual world.
19. The non-transitory computer-readable storage medium of claim 18 , wherein the instructions further cause the client device to perform an operation comprising:
sending, to a client device including the camera, a view of the location in the virtual world corresponding to the determined pose.
20. A computer system comprising:
one or more computer processors; and
a non-transitory computer-readable storage medium storing instructions for locating a camera in an environment, the instructions when executed by the one or more computer processors performing actions comprising:
comparing a feature identified in an image captured by the camera to a 3D map of the environment;
identifying a correspondence between the feature and a 3D line junction in the 3D map by iteratively projecting endpoints from 3D to 2D using respective predicted poses to identify a predicted pose whose calculation error is below a threshold; and
determining a pose of the camera within the environment using the correspondence.