Computer-vision based positioning for augmented reality navigation

1. A computer-implemented method for computer vision based positioning for Augmented Reality (AR) navigation, the method comprising:

receiving a live camera image of a scene, the live camera image captured by a device at a viewpoint;

performing computer vision scene understanding on the live camera image;

obtaining navigation instructions from a navigation system;

generating a navigation indication from the navigation instructions, wherein the navigation indication is a route overlay;

performing a computer vision-based positioning algorithm on the live camera image, comprising:

feature matching a plurality of features in successive frames of the live camera image using a machine learning model,

computing, based on the feature matching, an estimated motion of the viewpoint;

predicting the movement of one or more landmarks in the live camera image based on the estimated motion of the viewpoint, and

determining, based on the estimated motion and the actual movement of the landmarks in the live camera image, the relative position between the viewpoint and the landmarks;

generating a 3D model of a local environment around the device based on the computer vision-based positioning algorithm;

generating a visual display of the navigation indication;

transforming, via an affine transformation and based on the relative position between the viewpoint and the one or more landmarks determined via the computer vision-based positioning algorithm, the shape and size of the visual display of the navigation indication such that the visual display appears textured on a surface of the 3D model of the local environment;

determining coordinates in the live camera image corresponding to the transformed visual display of the navigation indication in the 3D model of the local environment; and

displaying the scene with an augmented reality overlay comprising the transformed visual display of the navigation indication appearing textured on a surface of the scene.

2. The method of claim 1 , wherein:

the navigation indication is displayed to appear on top of the ground.

3. The method of claim 1 , wherein:

the navigation indication is a world-locked point of interest.

4. The method of claim 1 , wherein:

the computer vision scene understanding identifies one or more semantic objects in the live camera image.

5. The method of claim 1 , wherein:

the computer vision-based positioning algorithm is visual odometry.

6. The method of claim 5 , further comprising:

receiving the successive frames of the live camera image;

detecting the plurality of features in one or more of the successive frames; and

performing local optimization of the estimated motion.

7. The method of claim 1 , wherein:

the computer vision-based positioning algorithm is visual inertial odometry.

8. The method of claim 7 , further comprising:

receiving the successive frames of the live camera image;

detecting the plurality of features in one or more of the successive frames;

reading data from an inertial measurement unit and using the data for computing the estimated motion; and

performing local optimization of the estimated motion.

9. The method of claim 1 , wherein:

the computer vision-based positioning algorithm is Simultaneous Localization and Mapping (SLAM).

10. The method of claim 9 , further comprising:

receiving the successive frames of the live camera image;

extracting, for each of the successive frames, one or more SLAM landmarks from the frame;

associating each SLAM landmark with one or more corresponding SLAM landmarks in prior frames of the live camera image;

storing the estimated location; and

updating one or more stored estimated locations of the SLAM landmarks,

wherein determining the relative position between the viewpoint and the landmarks comprises using the SLAM landmarks as the one or more landmarks in the live camera image.

11. A computer-implemented method for computer vision based positioning for Augmented Reality (AR) navigation, the method comprising:

receiving a live camera image of a scene, the live camera image captured by a device at a viewpoint;

scaling and transforming the live camera image for display on the device;

reading, by the device, a current Global Positioning System (GPS) location;

performing computer vision scene understanding on the live camera image to identify one or more semantic objects in the live camera image;

obtaining navigation instructions from a navigation system;

generating a route overlay and a point of interest from the navigation instructions;

performing a computer vision-based positioning algorithm on the live camera image, comprising:

feature matching a plurality of features in successive frames of the live camera image using a machine learning model,

computing, based on the feature matching, an estimated motion of the viewpoint;

predicting the movement of one or more landmarks in the live camera image based on the estimated motion of the viewpoint, and

determining, based on the estimated motion and the actual movement of the landmarks in the live camera image, the relative position between the viewpoint and the landmarks;

generating a 3D model of a local environment around the device based on the computer vision-based positioning algorithm;

generating a visual display of the route overlay;

transforming, via an affine transformation and based on the relative position between the viewpoint and the one or more landmarks determined via the computer vision-based position algorithm, the shape and size of the visual display of the route overlay such that the visual display appears textured on a surface of the 3D model of the local environment;

determining coordinates in the live camera image corresponding to the transformed visual display of the route overlay in the 3D model of the local environment; and

displaying the scene with an augmented reality overlay comprising the transformed visual display of the route overlay appearing textured on a surface of the scene and a visual display of the point of interest, wherein the location or shape of the visual display of the point of interest is transformed based on the computer vision-based positioning algorithm.

12. The method of claim 11 , wherein:

the shape of the visual display of the route overlay is transformed to appear on top of the ground.

13. The method of claim 11 , wherein:

the location of the visual display of the point of interest is transformed to appear locked to a location in a local environment.

14. The method of claim 11 , wherein:

the semantic objects comprise road lanes, moving objects, non-moving objects, environmental objects, and a horizon line.

15. The method of claim 11 , wherein:

the computer vision-based positioning algorithm is visual odometry.

16. The method of claim 11 , further comprising:

receiving the successive frames of the live camera image;

detecting the plurality of features in one or more of the successive frames; and

performing local optimization of the estimated motion.

17. The method of claim 11 , wherein:

the computer vision-based positioning algorithm is visual inertial odometry.

18. The method of claim 11 , further comprising:

receiving the successive frames of the live camera image;

detecting the plurality of features in one or more of the successive frames;

reading data from an inertial measurement unit and using the data for estimating motion; and

performing local optimization of the estimated motion.

19. The method of claim 11 , wherein:

the computer vision-based positioning algorithm is Simultaneous Localization and Mapping (SLAM).

20. The method of claim 19 , further comprising:

receiving the successive frames of the live camera image;

extracting, for each of the successive frames, one or more SLAM landmarks from the frame;

associating each SLAM landmark with one or more corresponding SLAM landmarks in prior frames of the live camera image;

storing the estimated location; and

updating one or more stored estimated locations of the SLAM landmarks,

wherein determining the relative position between the viewpoint and the landmarks comprises using the SLAM landmarks as the one or more landmarks in the live camera image.