Real-time monocular structure from motion

1. A system for autonomous vehicular navigation using a multithreaded monocular structure from motion (SFM) architecture, comprising:

a single camera positioned on-board a vehicle; and

a multithreaded processor coupled to the single camera and a non-transitory computer-readable storage medium, the multithreaded processor being included in the multithreaded SFM architecture, and being configured for:

acquiring images using the single camera positioned on-board the vehicle;

estimating camera motion using monocular SFM by performing continuous camera pose estimation of 2D-3D correspondences;

detecting a local planarity of a road and correcting for scale drift using the monocular SFM based on the camera pose estimation;

determining visual odometry correspondences and replenishing 3D points of the images by combining the pose estimation with a 2D-2D multi-threaded per-frame epipolar search, the per-frame epipolar search continuously generating, for each frame, redundantly validated 3D points persistent across comparatively long tracks, the determining visual odometry further comprising:

parallelizing the epipolar search across a plurality of threads;

validating the 3D points among intermediate points, performing a local bundle adjustment, and retaining the validated 3D points for insertion of new 3D points at a keyframe in a main thread to frontload the validated 3D points for the pose estimation;

performing, in real-time, the pose estimation at each of a plurality of frames using only the validated 3D points to reduce processing requirements and increase processing speed of the pose estimation;

executing a real-time global bundle adjustment in a thread-safe architecture in parallel with the real-time pose estimation;

determining an optimized planar homography mapping between a road in two frames based on one or more determined ground estimation cues, the cues including triangulated sparse 3D points, inter-frame dense stereo matching, and a displacement computed based on a distance from a vanishing point; and

estimating per-frame relative importances of the cues by computing observation covariances for each of the cues and performing ground plane estimation by combining the triangulated sparse 3D points and cues using a Kalman filter; and

autonomously controlling driving functions of the vehicle based on the determined visual odometry and the local planarity of the road.

2. The system of claim 1 , wherein the processor is further configured for providing fast 3D-2D correspondences using pose-guided matching.

3. The system of claim 1 , wherein the processor is further configured for performing epipolar constrained search to produce per-frame 2D-2D correspondences.

4. The system of claim 1 , wherein the processor is further configured for performing vanishing point detection to hypothesize a feature match search window along one or more radial lines from the VP for pruning mismatches due to repeated features.

5. The system of claim 1 , wherein the processor is further configured for validating each 3D point against all frames in real-time, refining cameras and 3D points by the performing the local bundle adjustment, and collecting and refining 3D points from an epipolar thread.

6. The system of claim 1 , wherein the processor is further configured for collecting and refinding to allow bundle adjustment using long tracks.

7. The system of claim 1 , wherein the processor is further configured for performing real-time scale correcting by combining scale estimates from 3D points and planar homography mappings.

8. The system of claim 1 , wherein the validating the 3D points provides increased accuracy and timing of the pose estimation based on the epipolar constrained search, triangulation and backprojection.

9. The system of claim 1 , wherein the data-driven covariance learning comprises adapting the observation covariance on a per-frame basis by combining the cues triangulated sparse 3D points and homography-guided dense inter-frame stereo cues using a Kalman filter, where relative importance of cues is estimated on a per-frame basis learned through observation covariances.

10. A method for vehicular navigation using a multithreaded monocular structure from motion (SFM) architecture, comprising:

autonomously navigating a vehicle using a multithreaded processor coupled to a non-transitory computer-readable storage medium, the multithreaded processor being included in the multithreaded SFM architecture, and being configured for:

acquiring images using a single camera positioned on-board a vehicle;

estimating camera motion using monocular SFM by performing continuous camera pose estimation of 2D-3D correspondences;

detecting a local planarity of a road and correcting for scale drift using the monocular SFM based on the camera pose estimation;

determining visual odometry correspondences and replenishing 3D points of the images by combining the pose estimation with a 2D-2D multi-threaded per-frame epipolar search, the per-frame epipolar search continuously generating, for each frame, redundantly validated 3D points persistent across comparatively long tracks, the determining visual odometry further comprising:

parallelizing the epipolar search across a plurality of threads;

validating the 3D points among intermediate points, performing a local bundle adjustment, and retaining the validated 3D points for insertion of new 3D points at a keyframe in a main thread to frontload the validated 3D points for the pose estimation;

performing, in real-time, the pose estimation at each of a plurality of frames using only the validated 3D points to reduce processing requirements and increase processing speed of the pose estimation;

executing a real-time global bundle adjustment in a thread-safe architecture in parallel with the real-time pose estimation;

determining an optimized planar homography mapping between a road in two frames based on one or more determined ground estimation cues, the cues including triangulated sparse 3D points, inter-frame dense stereo matching, and a displacement computed based on a distance from a vanishing point; and

estimating per-frame relative importances of the cues by computing observation covariances for each of the cues and performing ground plane estimation by combining the triangulated sparse 3D points and cues using a Kalman filter; and

autonomously controlling driving functions of the vehicle based on the determined visual odometry and the local planarity of the road.

11. The method of claim 10 , further comprising providing fast 3D-2D correspondences using pose-guided matching.

12. The method of claim 10 , further comprising performing epipolar constrained search to produce per-frame 2D-2D correspondences.

13. The method of claim 10 , further comprising performing vanishing point detection to hypothesize a feature match search window along one or more radial lines from the VP for pruning mismatches due to repeated features.

14. The method of claim 10 , further comprising validating each 3D point against all frames in real-time, refining cameras and 3D points by the performing the local bundle adjustment, and collecting and refining 3D points from an epipolar thread.

15. The method of claim 10 , further comprising collecting and refinding to allow bundle adjustment using long tracks.

16. The method of claim 10 , further comprising performing real-time scale correcting by combining scale estimates from 3D points and planar homography mappings.

17. The method of claim 10 , wherein the validating the 3D points provides increased accuracy and timing of the pose estimation based on the epipolar constrained search, triangulation and backprojection.

18. The method of claim 10 , wherein the data-driven covariance learning comprises adapting the observation covariance on a per-frame basis by combining the cues triangulated sparse 3D points and homography-guided dense inter-frame stereo cues using a Kalman filter, where relative importance of cues is estimated on a per-frame basis learned through observation covariances.