Visual camera re-localization using graph neural networks and relative pose supervision
The present disclosure describes approaches to camera re-localization using a graph neural network (GNN). A re-localization model includes encoding an input image into a feature map. The model retrieves reference images from an image database of a previously scanned environment based on the feature map of the image. The model builds a graph based on the image and the reference images, wherein nodes represent the image and the reference images, and edges are defined between the nodes. The model may iteratively refine the graph through auto-aggressive edge-updating and message passing between nodes. With the graph built, the model predicts a pose of the image based on the edges of the graph. The pose may be a relative pose in relation to the reference images, or an absolute pose.
1. A computer-implemented method for generating virtual content for display by a client device, the method comprising:
receiving an image of a scene captured by a camera of the client device;
inputting the image of the scene into a re-localization model, wherein the re-localization model performs steps comprising:
encoding the image into a feature map;
retrieving reference images from an image database based on the feature map of the image;
building a graph based on the image and the reference images comprising:
feature vector nodes generated from the feature map that represent the image and the reference images, and
edges between the feature vector nodes; and
predicting a pose of the image based on the graph;
receiving, from the re-localization model, the predicted pose for the image of the scene; and
generating, using the predicted pose, virtual content for display by the client device in conjunction with the scene.
2. The method of claim 1 , wherein encoding the image into a feature map comprises one or more of:
image convolution;
pattern recognition;
object recognition;
edge detection; and
semantic segmentation.
3. The method of claim 1 , wherein the feature map is a one-dimensional vector of features.
4. The method of claim 1 , wherein retrieving the reference images comprises:
identifying a most similar reference image having a second feature map that is most similar to the feature map of the image; and
identifying additional reference images having feature maps in proximity to the feature map of the most similar reference image.
5. The method of claim 1 , wherein building the graph comprises:
initializing the reference images and the image of the scene as nodes of the graph; and
initializing edges between the nodes of the graph.
6. The method of claim 5 , wherein initializing the reference images and the image of the scene as nodes comprises encoding the image and the reference images into node feature vectors.
7. The method of claim 6 , wherein initializing the edges comprises, for each edge between two nodes:
concatenating the node feature vectors of the two nodes; and
projecting the concatenated node feature vectors into an edge feature vector.
8. The method of claim 7 , wherein building the graph comprises updating edge feature vectors in an auto-regressive manner.
9. The method of claim 1 , wherein the pose comprises a relative pose in relation to the reference images.
10. The method of claim 9 , wherein the pose further comprises an absolute pose based on the relative pose and a ground truth pose of one of the reference images.
11. A non-transitory computer-readable storage medium storing instructions that, when executed by a computer processor of a client device, cause the computer processor to perform operations comprising:
receiving an image of a scene captured by a camera of the client device;
inputting the image of the scene into a re-localization model, wherein the re-localization model performs steps comprising:
encoding the image into a feature map;
retrieving reference images from an image database based on the feature map of the image;
building a graph based on the image and the reference images comprising:
feature vector nodes generated from the feature map that represent the image and the reference images, and
edges between the feature vector nodes; and
predicting a pose of the image based on the graph;
receiving, from the re-localization model, the predicted pose for the image of the scene; and
generating, using the predicted pose, virtual content for display by the client device in conjunction with the scene.
12. The non-transitory computer-readable storage medium of claim 11 , wherein the feature map is a one-dimensional vector of features.
13. The non-transitory computer-readable storage medium of claim 11 , wherein retrieving the reference images comprises:
identifying a most similar reference image having a second feature map that is most similar to the feature map of the image; and
identifying additional reference images having feature maps in proximity to the feature map of the most similar reference image.
14. The non-transitory computer-readable storage medium of claim 11 , wherein building the graph comprises:
initializing nodes of the graph by encoding the image and the reference images into node feature vectors; and
initializing edges between the nodes of the graph by:
concatenating the node feature vectors of the two nodes, and
projecting the concatenated feature vectors into an edge feature vector.
15. The non-transitory computer-readable storage medium of claim 14 , wherein building the graph comprises:
updating edge feature vectors in an auto-regressive manner;
for each edge, generating a first message from a first node to a second node based on the node feature vector of the first node and the edge feature vector and a second message from the second node to the first node based on the node feature vector of the second node and the edge feature vector; and
for each node, aggregating messages directed to the node from other nodes of the graph.
16. The non-transitory computer-readable storage medium of claim 11 , wherein predicting the pose of the image based on the graph comprises inputting the edges into a regression layer of the re-localization model to output the pose.
17. The non-transitory computer-readable storage medium of claim 11 , wherein the pose comprises a relative pose in relation to the reference images.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the pose further comprises an absolute pose based on the relative pose and a ground truth pose of one of the reference images.