Image based localization system
Systems and methods for determining a location based on image data are provided. A method can include receiving, by a computing system, a query image depicting a surrounding environment of a vehicle. The query image can be input into a machine-learned image embedding model and a machine-learned feature extraction model to obtain a query embedding and a query feature representation, respectively. The method can include identifying a subset of candidate embeddings that have embeddings similar to the query embedding. The method can include obtaining a respective feature representation for each image associated with the subset of candidate embeddings. The method can include determining a set of relative displacements between each image associated with the subset of candidate embeddings and the query image and determining a localized state of a vehicle based at least in part on the set of relative displacements.
1. A computer-implemented method for determining a location of a vehicle, the method comprising:
receiving image data associated with an environment of the vehicle;
processing the image data with a machine-learned image embedding model to generate a query embedding for the image data;
identifying one or more identified image embeddings of a plurality of image embeddings associated with the environment of the vehicle based on a plurality of similarity scores between the plurality of image embeddings and the query embedding, wherein one or more similarity scores of the plurality of similarity scores satisfy a threshold similarity score, the one or more similarity scores respective to the one or more identified image embeddings; and
determining the location of the vehicle based on a comparison between the query embedding and the one or more identified image embeddings of the plurality of image embeddings associated with the environment of the vehicle.
2. The computer-implemented method of claim 1 , wherein the plurality of image embeddings are previously computed for a plurality of images of the environment by the machine-learned image embedding model.
3. The computer-implemented method of claim 1 , wherein the one or more identified image embeddings associated with the environment of the vehicle are obtained from a feature representation database remotely located from the vehicle.
4. The computer-implemented method of claim 1 , further comprising:
obtaining the one or more identified image embeddings associated with the environment of the vehicle based on vehicle location data associated with the vehicle.
5. The computer-implemented method of claim 4 , wherein the vehicle location data comprises coarse geolocation coordinates.
6. The computer-implemented method of claim 5 , wherein the coarse geolocation coordinates comprise global positioning system coordinates.
7. The computer-implemented method of claim 4 , wherein the one or more identified image embeddings are associated with image location data, and wherein the one or more identified image embeddings associated with the environment of the vehicle are obtained based on a comparison between the vehicle location data and the image location data.
8. The computer-implemented method of claim 7 , wherein the location of the vehicle is determined based on the image location data.
9. The computer-implemented method of claim 1 , wherein the image data comprises a query image depicting at least a portion of a surrounding environment of the vehicle.
10. The computer-implemented method of claim 9 , wherein determining the location of the vehicle based on the comparison between the query embedding and the one or more identified image embeddings associated with the environment of the vehicle comprises:
determining a relative displacement between the query image and an image associated with at least one of the one or more identified image embeddings; and
determining the location of the vehicle based on the relative displacement.
11. The computer-implemented method of claim 1 , wherein the location of the vehicle is indicative of one or more current geolocation coordinates and a heading angle of the vehicle.
12. The computer-implemented method of claim 1 , wherein the image data is camera data, LIDAR data, or RADAR data.
13. A computing system comprising:
one or more processors; and
one or more tangible, non-transitory, computer readable media that store instructions for execution by the one or more processors to cause the computing system to perform operations, the operations comprising:
receiving image data associated with an environment of a vehicle;
processing the image data with a machine-learned image embedding model to generate a query embedding for the image data;
identifying one or more identified image embeddings of a plurality of image embeddings associated with the environment of the vehicle based on a plurality of similarity scores between the plurality of image embeddings and the query embedding, wherein one or more similarity scores of the plurality of similarity scores satisfy a threshold similarity score, the one or more similarity scores respective to the one or more identified image embeddings; and
determining a location of the vehicle based on a comparison between the query embedding and the one or more identified image embeddings of the plurality of image embeddings associated with the environment of the vehicle.
14. The computing system of claim 13 , wherein the plurality of image embeddings are previously computed for a plurality of images of the environment by the machine-learned image embedding model.
15. The computing system of claim 13 , further comprising:
obtaining the one or more identified image embeddings associated with the environment of the vehicle based on vehicle location data associated with the vehicle.
16. The computing system of claim 13 , wherein the image data comprises a query image depicting at least a portion of a surrounding environment of the vehicle.
17. The computing system of claim 16 , wherein the computing system is located onboard the vehicle, wherein the computing system comprises one or more cameras, and wherein the query image is collected by the one or more cameras.
18. The computing system of claim 13 , wherein the vehicle comprises an autonomous truck.
19. The computing system of claim 18 , wherein the operations further comprise:
controlling a motion of the autonomous truck based on the location of the vehicle.
20. One or more non-transitory, computer-readable media storing instructions that are executable by one or more processors to cause the one or more processors to perform operations, the operations comprising:
receiving image data associated with an environment of a vehicle;
processing the image data with a machine-learned image embedding model to generate a query embedding for the image data;
identifying one or more identified image embeddings of a plurality of image embeddings associated with the environment of the vehicle based on a plurality of similarity scores between the plurality of image embeddings and the query embedding, wherein one or more similarity scores of the plurality of similarity scores satisfy a threshold similarity score, the one or more similarity scores respective to the one or more identified image embeddings; and
determining a location of the vehicle based on a comparison between the query embedding and the one or more identified image embeddings of the plurality of image embeddings associated with the environment of the vehicle.