Deep direct localization from ground imagery and location readings

1. A method comprising, by a computing system:

receiving an image of a ground scene capturing an environment surrounding a vehicle operating in the environment, the image being captured by one or more sensors associated with the vehicle;

generating a feature representation of the image;

determining, based on the feature representation of the image, a potential ground control point corresponding to an object in the environment;

determining a predetermined location reading associated with the potential ground control point;

calculating a differential relative to the predetermined location reading associated with the potential ground control point; and

determining, based on the differential and the predetermined location reading associated with the potential ground control point, a location of the vehicle.

2. The method of claim 1 , further comprising:

calculating an estimated time of arrival of the vehicle to arrive at a destination location based on the determined location of the vehicle.

3. The method of claim 1 , further comprising:

generating an adjusted location reading by combining the differential and the predetermined location reading, wherein the adjusted location reading corresponds to the location of the vehicle.

4. The method of claim 1 , wherein calculating the differential is based on an image localization model trained based on a plurality of training images captured by one or more sensors associated with one or more vehicles.

5. The method of claim 4 , wherein calculating the differential comprises:

inputting the feature representation of the image and the predetermined location reading associated with the potential ground control point to the image localization model; and

outputting, by the image localization model, a set of two differential values, corresponding to latitude and longitude respectively.

6. The method of claim 5 , wherein determining the location of the vehicle comprises combining the predetermined location reading associated with the potential ground control point with the set of two differential values.

7. The method of claim 4 , wherein each of the plurality of training images is associated with one or more of a raw location reading or a ground control point with a predetermined location reading.

8. The method of claim 4 , further comprising training the image localization model by:

generating feature representations of the plurality of training images, wherein the plurality of training images are respectively associated with a plurality of raw location readings and a plurality of ground control points; and

learning a relationship between (1) the feature representations of the plurality of training images and (2) differentials between the plurality of raw location readings and a plurality of predetermined location readings for the plurality of ground control points.

9. The method of claim 8 , wherein the relationship comprises a mapping function, and wherein calculating the differential comprises:

inputting the feature representation of the image and the predetermined location reading based on the potential ground control point to the mapping function; and

outputting, by the mapping function, the differential of a predicted hypothesis location reading and the predetermined location reading, wherein the differential comprises a set of two differential values corresponding to latitude and longitude respectively.

10. The method of claim 4 , wherein the image localization model is based on a machine-learning architecture comprising a convolutional neural network, and wherein the feature representation of the image is processed by the convolutional neural network.

11. The method of claim 1 , wherein the predetermined location reading associated with the potential ground control point comprise one or more of a GPS coordinate, a location reading determined based on speed, a location reading determined based on bearing, a location reading determined based on gyroscope rotation data, or a location reading determined based on one or more reference maps.

12. A system comprising: one or more processors and one or more computer-readable non-transitory storage media coupled to the one or more processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by the one or more processors to cause the system to perform operations comprising:

receiving an image of a ground scene capturing an environment surrounding a vehicle operating in the environment, the image being captured by one or more sensors associated with the vehicle;

generating a feature representation of the image;

determining, based on the feature representation of the image, a potential ground control point corresponding to an object in the environment;

determining a predetermined location reading associated with the potential ground control point;

calculating a differential relative to the predetermined location reading associated with the potential ground control point; and

determining, based on the differential and the predetermined location reading associated with the potential ground control point, a location of the vehicle.

13. The system of claim 12 , wherein the one or more computer-readable non-transitory storage media comprise instructions further operable when executed by the one or more processors to cause the system to perform operations comprising calculating an estimated time of arrival of the vehicle to arrive at a destination location based on the determined location of the vehicle.

14. The system of claim 12 , wherein the one or more computer-readable non-transitory storage media comprise instructions further operable when executed by one or more of the processors to cause the system to perform operations comprising generating an adjusted location reading by combining the differential and the predetermined location reading, wherein the adjusted location reading corresponds to the location of the vehicle.

15. The system of claim 12 , wherein calculating the differential is based on an image localization model trained based on a plurality of training images captured by one or more sensors associated with one or more vehicles.

16. The system of claim 15 , wherein calculating the differential comprises:

inputting the feature representation of the image and the predetermined location reading associated with the potential ground control point to the image localization model; and

outputting, by the image localization model, a set of two differential values, corresponding to latitude and longitude respectively.

17. One or more computer-readable non-transitory storage media embodying instructions that are operable when executed to cause one or more processors to perform operations comprising:

receiving an image of a ground scene capturing an environment surrounding a vehicle operating in the environment, the image being captured by one or more sensors associated with the vehicle;

generating a feature representation of the image;

determining, based on the feature representation of the image, a potential ground control point corresponding to an object in the environment;

determining a predetermined location reading associated with the potential ground control point;

calculating a differential relative to the predetermined location reading associated with the potential ground control point; and

determining, based on the differential and the predetermined location reading associated with the potential ground control point, a location of the vehicle.

18. The media of claim 17 , wherein the instructions are further operable when executed to cause the one or more processors to perform operations comprising calculating an estimated time of arrival of the vehicle to arrive at a destination location based on the determined location of the vehicle.

19. The media of claim 17 , wherein the instructions are further operable when executed to cause the one or more processors to perform operations comprising generating an adjusted location reading by combining the differential and the predetermined location reading, wherein the adjusted location reading corresponds to the location of the vehicle.

20. The media of claim 17 , wherein calculating the differential is based on an image localization model trained based on a plurality of training images captured by one or more sensors associated with one or more vehicles.