Robust scale estimation in real-time monocular SFM for autonomous driving

1. A method for performing three-dimensional (3D) localization requiring only a single camera, comprising:

capturing images from only one camera;

generating a cue combination from sparse features, dense stereo and object bounding boxes;

correcting for scale in monocular structure from motion (SFM) using the cue combination for estimating a ground plane;

performing localization by combining SFM, ground plane and object bounding boxes to produce a 3D object localization, and

back-projecting through a ground plane {h,n}:

B

=

(

B

x

,

B

y

,

B

z

)

T

=

-

hK

-

1

⁢

b

n

T

⁢

K

-

1

⁢

b

,

where K is a camera intrinsic calibration matrix and b=(x, y, 1) T is a bottom of a 2D bounding box in homogeneous coordinates.

2. The method of claim 1 , comprising combining monocular real-time SFM, a cue combination and object tracking for 3D localization.

3. The method of claim 1 , comprising with tracked bounding boxes, determining regions of an image that are background (non-moving objects) and using monocular SFM to estimate a camera pose and the ground plane.

4. The method of claim 1 , comprising performing a dense optical flow estimation on the object.

5. The method of claim 1 , comprising estimating the ground plane using 3D points, dense stereo and 2D object bounding boxes.

6. The method of claim 1 , comprising learning one or more models that indicate per-frame relative importance of cues.

7. The method of claim 1 , comprising combining ground plane estimates within a Kalman filter.

8. The method of claim 1 , comprising applying an estimated ground plane to correct a monocular SFM scale drift.

9. The method of claim 8 , wherein the estimated ground plane is used to find a 3D bounding box that encloses the object.

10. The method of claim 1 , comprising performing autonomous driving and driving safety with the 3D localization.

11. A vehicle, comprising:

a single camera;

a motor coupled to the single camera for moving the vehicle; and

means for three-dimensional (3D) localization of traffic participants including vehicles or pedestrians, said means including:

means for generating a cue combination from sparse features, dense stereo and object bounding boxes;

means for correcting for scale in monocular structure from motion (SFM) using the cue combination for estimating a ground plane;

means for performing localization by combining SFM, ground plane and object bounding boxes to produce a 3D object localization, and back-projecting through a ground plane {h,n}:

B

=

(

B

x

,

B

y

,

B

z

)

T

=

-

hK

-

1

⁢

b

n

T

⁢

K

-

1

⁢

b

,

where K is a camera intrinsic calibration matrix and b=(x, y, 1) T is a bottom of a 2D bounding box in homogeneous coordinates.

12. The method of claim 1 , wherein methods i =1, . . . ,m are used to estimate a ground plane, with observation covariances U j , comprising determining fusion at time instant k as:

U

k

=

(

∑

i

=

1

m

⁢

(

U

i

k

)

-

1

)

-

1

,

z

k

=

U

k

⁢

∑

i

=

1

m

⁢

(

U

i

k

)

-

1

⁢

z

i

k

.

13. The method of claim 1 , comprising determining Plane-Guided Dense Stereo, including determing a region of interest (ROI) in a foreground corresponding to a planar ground.

14. The method of claim 13 , for a value of {h,n} and a relative camera pose {R, t} between frames k and k +1, comprising determining a per-pixel mapping using a homography matrix

G

=

R

+

1

h

⁢

tn

T

.

15. A method for performing three-dimensional (3D) localization of traffic participants including vehicles or pedestrians, requiring only a single camera, comprising:

capturing images from only one camera;

generating a cue combination from sparse features, dense stereo and object bounding boxes

correcting for scale in monocular structure from motion (SFM) using the cue combination for estimating a ground plane;

performing localization by combining SFM, ground plane and object bounding boxes to produce a 3D object localization, and

back-projecting through a ground plane {h,n}:

B

=

(

B

x

,

B

y

,

B

z

)

T

=

-

hK

-

1

⁢

b

n

T

⁢

K

-

1

⁢

b

,

where K is a camera intrinsic calibration matrix and b =(x, y, 1) T is a bottom of a 2D bounding box in homogeneous coordinates.

16. The method of claim 15 , comprising combining monocular real-time SFM, a cue combination and object tracking for 3D localization.

17. The method of claim 15 , comprising with tracked bounding boxes, determining regions of an image that are background (non-moving objects) and using monocular SFM to estimate a camera pose and the ground plane.

18. The method of claim 15 , comprising performing a dense optical flow estimation on the object.

19. The method of claim 15 , comprising estimating the ground plane using 3 D points, dense stereo and 2D object bounding boxes.

20. A method for performing three-dimensional (3D) localization requiring only a single camera, comprising:

capturing images from only one camera;

generating a cue combination from sparse features, dense stereo and object bounding boxes;

correcting for scale in monocular structure from motion (SFM) using the cue combination for estimating a ground plane;

performing localization by combining SFM, ground plane and object bounding boxes to produce a 3D object localization

wherein methods i =1, . . . ,m are used to estimate a ground plane, with observation covariances U j , comprising determining fusion at time instant k as:

U

k

=

(

∑

i

=

1

m

⁢

(

U

i

k

)

-

1

)

-

1

,

z

k

=

U

k

⁢

∑

i

=

1

m

⁢

(

U

i

k

)

-

1

⁢

z

i

k

.

21. A method for performing three-dimensional (3D) localization of traffic participants including vehicles or pedestrians, requiring only a single camera, comprising:

capturing images from only one camera;

generating a cue combination from sparse features, dense stereo and object bounding boxes

correcting for scale in monocular structure from motion (SFM) using the cue combination for estimating a ground plane; and

performing localization by combining SFM, ground plane and object bounding boxes to produce a 3D object localization, wherein methods i =1, . . . ,m are used to estimate a ground plane, with observation covariances U j , comprising determining fusion at time instant k as:

U

k

=

(

∑

i

=

1

m

⁢

(

U

i

k

)

-

1

)

-

1

,

z

k

=

U

k

⁢

∑

i

=

1

m

⁢

(

U

i

k

)

-

1

⁢

z

i

k

.

22. A vehicle, comprising:

a single camera;

a motor coupled to the single camera for moving the vehicle; and

means for for three-dimensional (3D) localization of traffic participants including vehicles or pedestrians, said means including:

means for generating a cue combination from sparse features, dense stereo and object bounding boxes;

means for correcting for scale in monocular structure from motion (SFM) using the cue combination for estimating a ground plane;

means for performing localization by combining SFM, ground plane and object bounding boxes to produce a 3D object localization, and

wherein methods i =1, . . . ,m are used to estimate a ground plane, with observation covariances U j , comprising determining fusion at time instant k as:

U

k

=

(

∑

i

=

1

m

⁢

(

U

i

k

)

-

1

)

-

1

,

z

k

=

U

k

⁢

∑

i

=

1

m

⁢

(

U

i

k

)

-

1

⁢

z

i

k

.