Method and system for automatically detecting multi-object anomalies utilizing joint sparse reconstruction model
View Patent ↗Methods and systems for automatically detecting multi-object anomalies at a traffic intersection utilizing a joint sparse reconstruction model. A first input video sequence at a first traffic location can be received and at least one normal event involving P moving objects (where P is greater than or equal to 1) can be identified in an offline training phase. The normal event in the first input video sequence can be assigned to at least one normal event class and a training dictionary suitable for joint sparse reconstruction can be built in the offline training phase. A second input video sequence captured at a second traffic location similar to the first traffic location can be received and at least one event involving P moving objects can be identified in an online detection phase.
1. A method for detecting multi-object anomalies in transportation related video footage, said method comprising:
receiving in an offline training phase a first input video sequence at a first traffic location and identifying at least one normal event involving P moving objects, where P is greater than 1;
assigning in said offline training phase said at least one normal event in said first input video sequence to at least one normal event class and building a training dictionary suitable for joint sparse reconstruction;
receiving in an online detection phase a second input video sequence captured at a second traffic location similar to said first traffic location and identifying at least one event involving P moving objects;
reconstructing in said online detection phase an approximation of said event within second input video sequence with respect to said training dictionary using a joint sparse reconstruction model; and
determining in said online detection phase whether said event within second input video sequence is anomalous by evaluating an outlier rejection measure of said approximation and comparing said measure against a predetermined threshold, wherein said outlier rejection measure is given by
JSCI
(
S
′
)
=
K
·
max
i
λ
i
(
S
′
)
row
,
0
/
S
′
row
,
0
-
1
K
-
1
,
where
S
′
=
[
α
1
,
1
α
2
,
1
α
1
,
2
α
2
,
2
]
and α i,j are coefficient sub-vectors corresponding to coefficient vectors α i , where i=1, 2, . . . , P represents concatenation of sub-dictionaries from all classes belonging to an i-th trajectory and j represents a given class, K represents a number of normal event classes, λ i (S′) represents a characteristic function whose only non-zero entries are the rows in S′ that are associated with the i-th class, and row norm ∥ ∥ row,0 represents the number of non-zero rows of a matrix.
2. The method of claim 1 wherein P is 2.
3. The method of claim 1 , wherein an event involving said P moving objects is defined as a collection of P trajectories corresponding to said moving objects, wherein each trajectory is defined as a vector of n spatial coordinates of the object's path indexed by time.
4. The method of claim 3 where said training dictionary is defined as A=[A 1 , A 2 , . . . , A p ] where A i =[A i,1 , A i,2 , . . . , A i,K ]ε n×KT , i=1, 2, . . . , P represents concatenation of sub-dictionaries from all classes belonging to an i-th trajectory, K represents a number of normal event classes, and T represents a number of events per class.
5. The method of claim 4 wherein said joint sparse reconstruction model is given by: Y=AS=[A 1,1 , A 1,2 , . . . , A 1,K , . . . , A i,1 , A i,2 , . . . , A i,K , . . . , A p,1 , A p,2 , . . . , A p,K ][a 1 , . . . , a i , . . . , a p ]Y=AS wherein Y represents a P-object event in the second video input sequence, A represent said training dictionary, and S represent a sparse matrix of coefficient weights determined by solving an optimization problem.
6. The method of claim 5 wherein said optimization problem is given by
minimize
=
J
(
H
∘
S
)
row
,
0
subject
to
Y
-
AS
F
<
ɛ
where
H
=
[
1
0
…
0
0
1
…
0
⋮
⋮
⋮
⋮
0
0
…
1
]
,
J
=
[
I
KT
,
I
KT
,
…
,
I
KT
]
,
where operator ∘ indicates an entry-wise matrix Hadamard product, the vectors 1 and 0 in H are in R KT , and contain all ones and zeros respectively, the elements I KT in J are the KT-dimensional identity matrix, ∥ ∥ row,0 represents a row norm indicating the number of non-zero rows in said matrix, and ∥ ∥ F indicates said Frobenius norm.
7. The method of claim 6 wherein said optimization problem is solved by formulating an auxiliary convex optimization problem which minimizes Frobenius norm ∥Y−AS∥ F subject to a sparsity constraint defined as a linear membership constraint utilizing a simultaneous orthogonal matching pursuit algorithm.
8. The method of claim 1 further comprising constructing a plurality of training dictionaries indexed by one or more properties such as date, time, location, and retrieving in said online detection phase a most relevant training dictionary based on said properties.
9. The method of claim 1 further comprising:
identifying in said offline training phase at least one anomalous event in said first video sequence involving P moving objects, assigning said event to an anomalous event class, and adding said anomalous event class to said training dictionary;
assigning in said online detection phase said event within second input video sequence to one of the event classes in said training dictionary by minimizing a reconstruction error; and
determining in said online detection phase that said event within second input video sequence is anomalous if it is assigned to an anomalous event class.
10. The method of claim 9 wherein said assignment in online detection phase is given by identity
(
Y
)
=
arg
min
i
Y
-
A
δ
i
(
S
)
F
wherein δ i (S) represents said matrix whose only nonzero entries are similar as those in S associated with class I and ∥ ∥ F is the Frobenius norm.
11. A system for detecting multi-object anomalies in transportation related video footage, said system comprising:
a processor;
a data bus coupled to said processor; and
a computer-usable medium embodying computer program code, said computer-usable medium being coupled to said data bus, said computer program code comprising instructions executable by said processor and configured for:
receiving in an offline training phase a first input video sequence at a first traffic location and identifying at least one normal event involving P moving objects, where P is greater than 1;
assigning in said offline training phase said at least one normal event in said first input video sequence to at least one normal event class and building a training dictionary suitable for joint sparse reconstruction;
receiving in an online detection phase a second input video sequence captured at a second traffic location similar to said first traffic location and identifying at least one event involving P moving objects;
reconstructing in said online detection phase an approximation of said event within second input video sequence with respect to said training dictionary using a joint sparse reconstruction model; and
determining in said online detection phase whether said event within second input video sequence is anomalous by evaluating an outlier rejection measure of said approximation and comparing said measure against a predetermined threshold, wherein said outlier rejection measure is given by
JSCI
(
S
′
)
=
K
·
max
i
λ
i
(
S
′
)
row
,
0
/
S
′
row
,
o
-
1
K
-
1
,
where
S
′
=
[
α
1
,
1
α
2
,
1
α
1
,
2
α
2
,
2
]
and α i,j are coefficient sub-vectors corresponding to coefficient vectors α i , where i=1, 2, . . . , P represents concatenation of sub-dictionaries from all classes belonging to an i-th trajectory and j represents a given class, K represents a number of normal event classes, λ i (S′) represents a characteristic function whose only non-zero entries are the rows in S′ that are associated with the i-th class, and row norm ∥ ∥ row,0 represents the number of non-zero rows of a matrix.
12. The system of claim 11 wherein P is 2.
13. The system of claim 11 wherein an event involving said P moving objects is defined as a collection of P trajectories corresponding to said moving objects, wherein each trajectory is defined as a vector of n spatial coordinates of the object's path indexed by time.
14. The system of claim 11 where said training dictionary is defined as A=[A 1 , A 2 , . . . , A p ] where A i =[A i,1 , A i,2 , . . . A i,K ]ε n×KT , i=1, 2, . . . , P represents concatenation of sub-dictionaries from all classes belonging to an i-th trajectory, K represents a number of normal event classes, and T represents a number of events per class.
15. The system of claim 14 wherein said joint sparse reconstruction model is given by Y=AS=[A 1,1 , A 1,2 , . . . , A 1,K , . . . , A i,1 , A i,2 , . . . , A i,K , . . . , A p,1 , A p,2 , . . . , A p,K ][a 1 , . . . , a i , . . . , a p ]Y=AS wherein Y represents a P-object event in the second video input sequence, A represents said training dictionary, and S represents a sparse matrix of coefficient weights determined by solving an optimization problem.
16. A non-transitory processor-readable medium storing code representing instructions to cause a process for detecting multi-object anomalies in transportation related video footage, said code comprising code to:
receive in an offline training phase a first input video sequence at a first traffic location and identifying at least one normal event involving P moving objects, where P is greater than 1;
assign in said offline training phase said at least one normal event in said first input video sequence to at least one normal event class and building a training dictionary suitable for joint sparse reconstruction;
receive in an online detection phase a second input video sequence captured at a second traffic location similar to said first traffic location and identifying at least one event involving P moving objects;
reconstruct in said online detection phase an approximation of said event within second input video sequence with respect to said training dictionary using a joint sparse reconstruction model; and
determine in said online detection phase whether said event within second input video sequence is anomalous by evaluating an outlier rejection measure of said approximation and comparing said measure against a predetermined threshold, wherein said outlier rejection measure is given by
JSCI
(
S
′
)
=
K
·
max
i
λ
i
(
S
′
)
row
,
0
/
S
′
row
,
o
-
1
K
-
1
,
where
S
′
=
[
α
1
,
1
α
2
,
1
α
1
,
2
α
2
,
2
]
and α i,j are coefficient sub-vectors corresponding to coefficient vectors α i , where i=1, 2, . . . , P represents concatenation of sub-dictionaries from all classes belonging to an i-th trajectory and j represents a given class, K represents a number of normal event classes, λ i (S′) represents a characteristic function whose only non-zero entries are the rows in S′ that are associated with the i-th class, and row norm ∥ ∥ row,0 represents the number of non-zero rows of a matrix.
17. The non-transitory processor-readable medium of claim 16 wherein said code further comprises code to determine in said online detection phase whether said event within second input video sequence is anomalous by evaluating an outlier rejection measure of said approximation and comparing said measure against a predetermined threshold.