Method and system for moving object detection with single camera
Disclosed are systems and methods for detecting moving objects. A computer-implemented method for detecting moving objects comprises obtaining a streaming video captured by a camera; extracting an input image sequence including a series of images from the streaming video; tracking point features and maintaining a set of point trajectories for at least one of the series of images; measuring a likelihood for each point trajectory to determine whether it belongs to a moving object using constraints from multi-view geometry; and determining a conditional random field (CRF) on an entire frame to obtain a moving object segmentation.
1. A computer-implemented method for detecting moving objects, comprising:
obtaining a streaming video captured by a camera;
extracting an input image sequence including a series of images from the streaming video;
tracking point features and maintaining a set of point trajectories for at least one of the series of images;
measuring a likelihood for each point trajectory to determine whether it belongs to a moving object using constraints from multi-view geometry;
determining a conditional random field (CRF) on an entire frame to obtain a moving object segmentation,
wherein the method further comprises, for each frame T:
computing an optical flow and a point trajectory;
estimating fundamental matrices and a trifocal tensor;
computing an epipolar moving objectness score and a trifocal moving objectness score for each trajectory; and
forming the CRF on superpixels to determine moving labels.
2. The method of claim 1 , wherein the camera comprises a monocular camera.
3. The method of claim 1 , wherein the constraints from multi-view geometry comprise at least one of: an epipolar constraint between two-view and trifocal constraints from three-view.
4. The method of claim 3 , wherein the epipolar constraint is calculated based at least in part on an epipolar moving objectness score for a pair of point correspondence as follows:
ϕ( x i τ ,x i τ′ )= d pl ( F τ τ′ x i τ′ ,x i τ )+ d pl ( x i τ′ ,x i τ F τ τ′ ),
where F τ τ′ x i τ′ and x i τ F τ τ′ define relative epipolar lines in each view and a function d pl (·) computes a point to line distance.
5. The method of claim 4 , wherein the pair of point correspondence is determined based on an optical flow between consecutive frames.
6. The method of claim 4 , further comprising determining a weighted average of all epipolar moving objectness scores of a trajectory as follows:
Φ
(
z
i
,
τ
t
)
=
1
B
(
z
i
,
τ
t
)
∑
m
=
τ
+
1
t
β
t
-
m
∑
n
=
τ
m
-
1
ϕ
(
x
i
m
,
x
i
n
)
,
where β is a decay factor.
7. The method of claim 4 , further comprising determining another epipolar moving objectness score as follows:
Ψ
(
z
i
,
τ
t
)
=
1
ρ
(
z
i
,
τ
t
)
∑
m
=
τ
+
1
t
∑
n
=
τ
m
-
1
[
ϕ
(
x
i
m
,
x
i
n
)
>
ϕ
(
x
i
m
,
x
i
n
+
1
)
+
v
]
∑
m
=
τ
t
-
1
∑
n
=
m
t
[
ϕ
(
x
i
m
,
x
i
n
)
>
ϕ
(
x
i
m
+
1
,
x
i
n
)
+
v
]
,
where [·] is an indicator function, v is a tolerance variable and ρ(z i,τ t )=(t−τ)(t−τ−1)/2 is the normalization factor.
8. The method of claim 3 , wherein the trifocal constraints from three-view are determined based at least in part on a trifocal moving objectness score defined as follows:
γ( x i m ,x i n ,x i p )= d pp ( x i p ,{circumflex over (x)} i p )
where {circumflex over (x)} t″ is the estimated location in the third view using x t and x t′ and d pp (·) computes the Euclidean distance.
9. The method of claim 8 , further comprising determining a weighted average of all trifocal moving objectness scores to account for a history for each trajectory as follows:
Γ
(
z
i
,
τ
t
)
=
∑
m
=
τ
+
2
η
t
β
t
-
m
γ
(
x
i
m
,
x
i
n
,
x
i
p
)
∑
m
=
τ
+
2
η
T
β
t
-
m
,
where β is a decay factor.
10. A system for detecting moving objects, the system comprising:
a camera;
a memory; and
a processor communicatively coupled with the memory and the camera, the processor configured to:
obtain a streaming video captured by the camera;
extract an input image sequence including a series of images from the streaming video;
track point features and maintain a set of point trajectories for at least one of the series of images;
measure a likelihood for each point trajectory to determine whether it belongs to a moving object using constraints from multi-view geometry; and
determine a conditional random field (CRF) on an entire frame to obtain a dense moving object segmentation,
wherein the processor is further configured to for each frame T:
compute an optical flow and a point trajectory;
estimate fundamental matrices and a trifocal tensor;
compute an epipolar moving objectness score and a trifocal moving objectness score for each trajectory; and
form the CRF on superpiexels to determine moving labels.
11. The system of claim 10 , wherein the constraints from multi-view geometry comprise at least one of: an epipolar constraint between two-view and trifocal constraints from three-view.
12. The system of claim 11 , wherein the epipolar constraint is calculated based at least in part on an epipolar moving objectness score for a pair of point correspondence as follows:
ϕ( x i τ ,x i τ′ )= d pl ( F τ τ′ x i τ′ ,x i τ )+ d pl ( x i τ′ ,x i τ F τ τ′ ),
where F τ τ′ x i τ′ and x i τ F τ τ′ define relative epipolar lines in each view and a function d pl (·) computes a point to line distance.
13. The system of claim 12 , wherein the processor is configured to determine a weighted average of all epipolar moving objectness scores of a trajectory as follows:
Φ
(
z
i
,
τ
t
)
=
1
B
(
z
i
,
τ
t
)
∑
m
=
τ
+
1
t
β
t
-
m
∑
n
=
τ
m
-
1
ϕ
(
x
i
m
,
x
i
n
)
,
where β is a decay factor.
14. The system of claim 12 , wherein the processor is further configured to determine another epipolar moving objectness score as follows:
Ψ
(
z
i
,
τ
t
)
=
1
ρ
(
z
i
,
τ
t
)
∑
m
=
τ
+
1
t
∑
n
=
τ
m
-
1
[
ϕ
(
x
i
m
,
x
i
n
)
>
ϕ
(
x
i
m
,
x
i
n
+
1
)
+
v
]
∑
m
=
τ
t
-
1
∑
n
=
m
t
[
ϕ
(
x
i
m
,
x
i
n
)
>
ϕ
(
x
i
m
+
1
,
x
i
n
)
+
v
]
,
where [·] is an indicator function, visa tolerance variable and ρ(z i,τ t =(t−τ)(t−τ−1)/2 is the normalization factor.
15. The system of claim 11 , wherein the trifocal constraints from three-view is determined based at least in part on a trifocal moving objectness score as follows:
γ( x i m ,x i n ,x i p )= d pp ( x i p ,{circumflex over (x)} i p ),
where {circumflex over (x)} t″ is the estimated location in the third view using x t and x t′ and d pp (·) computes the Euclidean distance.
16. The system of claim 15 , wherein the processor is configured to determine a weighted average of all trifocal moving objectness scores to account for a history for each trajectory as follows:
Γ
(
z
i
,
τ
t
)
=
∑
m
=
τ
+
2
η
t
β
t
-
m
γ
(
x
i
m
,
x
i
n
,
x
i
p
)
∑
m
=
τ
+
2
η
T
β
t
-
m
,
where β is a decay factor.
17. A non-transitory computer readable storage medium having control logic stored thereon for causing a computer to detect moving objects, the control logic comprising machine executable codes for:
obtaining a streaming video captured by a camera;
extracting an input image sequence including a series of images from the streaming video;
tracking point features and maintaining a set of point trajectories for at least one of the series of images;
measuring a likelihood for each point trajectory to determine whether it belongs to a moving object using constraints from multi-view geometry; and
determining a conditional random field (CRF) on an entire frame to obtain a dense moving object segmentation,
further comprising, for each frame T, codes for:
computing an optical flow and a point trajectory;
estimating fundamental matrices and a trifocal tensor;
computing an epipolar moving objectness score and a trifocal moving objectness score for each trajectory; and
forming the CRF on superpiexels to determine moving labels.