Relocalization systems and methods
A method of determining a pose of an image capture device includes capturing an image using an image capture device. The method also includes generating a data structure corresponding to the captured image. The method further includes comparing the data structure with a plurality of known data structures to identify a most similar known data structure. Moreover, the method includes reading metadata corresponding to the most similar known data structure to determine a pose of the image capture device.
1. A method of determining a pose of an image capture device, comprising:
capturing an image using an image capture device;
mapping the captured image to an N dimensional vector using a neural network;
comparing the N dimensional vector with the plurality of known N dimensional vectors to identify a most similar known N dimensional vector; and
reading metadata corresponding to the most similar known N dimensional vector to determine a pose of the image capture device;
wherein the neural network is trained by mapping a plurality of known images to a plurality of known N dimensional vectors, wherein training the neural network comprises:
decreasing a first Euclidean distance between first and second known N dimensional vectors respectively corresponding to matching first and second known images in an N dimensional space; and
increasing a second Euclidean distance between first and third known N dimensional vectors respectively corresponding to non-matching first and third known images in the N dimensional space.
2. The method of claim 1 , wherein the N dimensional vector is a compact representation of the captured image.
3. The method of claim 2 , wherein the neural network is a convolutional neural network.
4. The method of claim 2 , wherein the N dimensional vector is a 128 dimensional vector.
5. The method of claim 2 , wherein comparing the N dimensional vector with the plurality of known N dimensional vectors to identify the most similar known N dimensional vector comprises:
determining respective Euclidean distances between the N dimensional vector and each respective known N dimensional vector; and
identifying a known N dimensional vector having a smallest distance to the N dimensional vector as the most similar known N dimensional vector.
6. The method of claim 2 , wherein each known image of the plurality has respective metadata, including pose data, and
wherein training the neural network comprises accessing a database of the known images annotated with the respective metadata.
7. The method of claim 6 , wherein the pose data encodes a translation and a rotation of a camera corresponding to a known image.
8. The method of claim 2 , wherein training the neural network comprises modifying the neural network based on comparing a triplet of known images of the plurality.
9. The method of claim 8 , wherein a first known image of the triplet is a matching image for a second known image of the triplet, and
wherein a third known image of the triplet is a non-matching image for the first known image of the triplet.
10. The method of claim 2 , wherein a first Euclidean distance between respective first and second pose data corresponding to the matching first and second known images is less than a predefined threshold, and
wherein a second Euclidean distance between respective first and third pose data corresponding to the non-matching first and third known images is more than the predefined threshold.
11. The method of claim 2 , wherein the metadata corresponding to the most similar known N dimensional vector includes pose data corresponding to the most similar known N dimensional vector, the method further comprising
determining a pose of the image capture device from the pose data in the metadata corresponding to the most similar known N dimensional vector.
12. A system for determining a pose of an image capture device, comprising:
an image capture device to capture an image;
a neural network to map the captured image to an N dimensional vector;
a processor to:
compare the N dimensional vector with the plurality of known N dimensional vectors to identify a most similar known N dimensional vector; and
read metadata corresponding to the most similar known N dimensional vector to determine a pose of the image capture device;
wherein the neural network is trained by mapping a plurality of known images to a plurality of known N dimensional vectors, wherein training the neural network comprises:
decreasing a first Euclidean distance between first and second known N dimensional vectors respectively corresponding to matching first and second known images in an N dimensional space; and
increasing a second Euclidean distance between first and third known N dimensional vectors respectively corresponding to non-matching first and third known images in the N dimensional space.
13. The system of claim 12 , wherein the N dimensional vector is a compact representation of the captured image.
14. The system of claim 13 , wherein the neural network is a convolutional neural network.
15. The system of claim 13 , wherein the N dimensional vector is a 128 dimensional vector.
16. The system of claim 13 , wherein the processor comparing the N dimensional vector with the plurality of known N dimensional vectors to identify the most similar known N dimensional vector comprises the processor:
determining respective Euclidean distances between the N dimensional vector and each respective known N dimensional vector; and
identifying a known N dimensional vector having a smallest distance to the N dimensional vector as the most similar known N dimensional vector.
17. The system of claim 13 , wherein each known image of the plurality has respective metadata, including pose data, and
wherein training the neural network comprises accessing a database of the known images annotated with the respective metadata.
18. The system of claim 13 , wherein training the neural network comprises modifying the neural network based on comparing a triplet of known images of the plurality,
wherein a first known image of the triplet is a matching image for a second known image of the triplet, and
wherein a third known image of the triplet is a non-matching image for the first known image of the triplet.
19. The system of claim 13 , wherein a first Euclidean distance between respective first and second pose data corresponding to the matching first and second known images is less than a predefined threshold, and
wherein a second Euclidean distance between respective first and third pose data corresponding to the non-matching first and third known images is more than the predefined threshold.
20. The system of claim 13 , wherein the metadata corresponding to the most similar known N dimensional vector includes pose data corresponding to the most similar known N dimensional vector, the system further comprising
determining a pose of the image capture device from the pose data in the metadata corresponding to the most similar known N dimensional vector.