Inter-trajectory anomaly detection using adaptive voting experts in a video surveillance system
View Patent ↗A sequence layer in a machine-learning engine configured to learn from the observations of a computer vision engine. In one embodiment, the machine-learning engine uses the voting experts to segment adaptive resonance theory (ART) network label sequences for different objects observed in a scene. The sequence layer may be configured to observe the ART label sequences and incrementally build, update, and trim, and reorganize an ngram trie for those label sequences. The sequence layer computes the entropies for the nodes in the ngram trie and determines a sliding window length and vote count parameters. Once determined, the sequence layer may segment newly observed sequences to estimate the primitive events observed in the scene as well as issue alerts for inter-sequence and intra-sequence anomalies.
1. A computer-implemented method for analyzing a scene depicted in an input stream of video frames captured by a video camera of a video surveillance system, the method comprising:
retrieving a first sequence and a second sequence, each providing an ordered string of labels, wherein each label corresponds to a cluster in an adaptive resonance theory (ART) network, wherein the strings of labels are generated by mapping kinematic data vectors generated for a first foreground object and a second foreground object detected in the input stream of video frames, respectively, to nodes of a self-organizing map (SOM) and clustering the nodes of the SOM using the ART network, and wherein the first sequence and the second sequence correspond to an observed interaction between the first foreground object and the second foreground object;
identifying one or more segments in each of the first and second sequences, wherein each segment includes a subsequence of the ordered string of labels in the first and second sequences;
determining a probability of observing the interaction between the first foreground object and the second foreground object, relative to a probability distribution generated from an ngram trie, wherein the ngram trie is generated from a plurality of previously observed sequences, each storing an ordered string of labels assigned to clusters in the ART network for objects detected in the input stream of video frames; and
upon determining the probability the observed interaction between the first foreground object and the second foreground object falls below a specified threshold, issuing an alert to a user of the video surveillance system.
2. The computer-implemented method of claim 1 , wherein identifying one or more segments in the each of the first and second sequences, comprises:
evaluating the first sequence and second sequence, respectively, using a sliding window of length (L), wherein an internal entropy expert and a boundary entropy expert each vote a position in each sliding window subsequence at which to split the sliding window sequence; and
inducing a segment boundary at each position in the first sequence and the second sequence that receives at least a number of votes satisfying a vote count threshold (V).
3. The computer-implemented method of claim 2 , wherein:
the internal entropy expert votes to minimize internal entropy by selecting a position which minimizes internal entropy in each respective sliding window sequence, based on the determined entropies; and
the boundary entropy expert votes by selecting a position in each respective sliding window that maximizes entropy between segments in the sliding window subsequence.
4. The computer-implemented method of claim 2 , further comprising, determining the sliding window length (L) as
L
=
⌈
arg
max
(
H
^
I
)
+
arg
max
(
H
^
B
)
2
⌉
,
wherein Ĥ I and Ĥ B , represent a standardized entropy value for the entropy measure determined for each node in the ngram trie and argmax( ) returns a level of the node in the trie with the greatest entropy values for Ĥ I and Ĥ B .
5. The computer-implemented method of claim 2 , wherein the vote count threshold (v) is determined as follows
V
=
⌈
∑
i
=
1
n
arg
max
(
H
^
I
)
i
+
arg
max
(
H
^
B
)
i
2
n
⌉
,
wherein Ĥ I and Ĥ B , represent a standardized entropy value for the entropy measure determined for each node in the ngram trie, wherein n represents a total number of branches in the ngram trie and argmax( ) returns a level of the node in the trie with the greatest entropy values for Ĥ I and Ĥ B .
6. The computer-implemented method of claim 1 , wherein determining the probability of the observed interaction between the first foreground object and the second foreground object, comprises determining a probability of observing a first segment associated with the first foreground object prior to the interaction and observing a second segment associated with the first foreground object following the interaction.
7. The computer-implemented method of claim 1 , wherein determining the probability of the observed interaction between the first foreground object and the second foreground object, comprises determining a probability of observing a first segment associated with the first foreground object prior to the interaction and observing a second segment associated with the second foreground object prior to the interaction.
8. The computer-implemented method of claim 1 , wherein determining the probability of the observed interaction between the first foreground object and the second foreground object, comprises determining a probability of observing a first segment associated with the first foreground object following the interaction and observing a second segment associated with the second foreground object following the interaction.
9. A non-transitory computer-readable medium containing a program, which when executed on a processor, performs an operation for analyzing a scene depicted in an input stream of video frames captured by a video camera of a video surveillance system, the operation comprising:
retrieving a first sequence and a second sequence, each providing an ordered string of labels, wherein each label corresponds to a cluster in an adaptive resonance theory (ART) network, wherein the strings of labels are generated by mapping kinematic data vectors generated for a first foreground object and a second foreground object detected in the input stream of video frames, respectively, to nodes of a self-organizing map (SOM) and clustering the nodes of the SOM using the ART network, and wherein the first sequence and the second sequence correspond to an observed interaction between the first foreground object and the second foreground object;
identifying one or more segments in each of the first and second sequences, wherein each segment includes a subsequence of the ordered string of labels in the first and second sequences;
determining a probability of an observed interaction between the first foreground object and the second foreground object, relative to a probability distribution generated from an ngram trie, wherein the ngram trie is generated from a plurality of previously observed sequences, each storing an ordered string of labels assigned to clusters in the ART network for objects detected in the input stream of video frames; and
upon determining the probability the observed interaction between the first foreground object and the second foreground object falls below a specified threshold, issuing an alert to a user of the video surveillance system.
10. The non-transitory computer-readable medium of claim 9 , wherein identifying one or more segments in the each of the first and second sequences, comprises:
evaluating the first sequence and second sequence, respectively, using a sliding window of length (L), wherein an internal entropy expert and a boundary entropy expert each vote a position in each sliding window subsequence at which to split the sliding window sequence; and
inducing a segment boundary at each position in the first sequence and the second sequence that receives at least a number of votes satisfying a vote count threshold (V).
11. The non-transitory computer-readable medium of claim 10 , wherein:
the internal entropy expert votes to minimize internal entropy by selecting a position which minimizes internal entropy in each respective sliding window sequence, based on the determined entropies; and
the boundary entropy expert votes by selecting a position in each respective sliding window that maximizes entropy between segments in the sliding window subsequence.
12. The non-transitory computer-readable medium of claim 10 , further comprising, determining the sliding window length (L) as
L
=
⌈
arg
max
(
H
^
I
)
+
arg
max
(
H
^
B
)
2
⌉
,
wherein Ĥ I and Ĥ B , represent a standardized entropy value for the entropy measure determined for each node in the ngram trie and argmax( ) returns a level of the node in the trie with the greatest entropy values for Ĥ I and Ĥ B .
13. The non-transitory computer-readable medium of claim 10 , wherein the vote count threshold (v) is determined as follows
V
=
⌈
∑
i
=
1
n
arg
max
(
H
^
I
)
i
+
arg
max
(
H
^
B
)
i
2
n
⌉
,
wherein Ĥ I and Ĥ B , represent a standardized entropy value for the entropy measure determined for each node in the ngram trie, wherein n represents a total number of branches in the ngram trie and argmax( ) returns a level of the node in the trie with the greatest entropy values for Ĥ I and Ĥ B .
14. The non-transitory computer-readable medium of claim 9 , wherein determining the probability of the observed interaction between the first foreground object and the second foreground object, comprises determining a probability of observing a first segment associated with the first foreground object prior to the interaction and observing a second segment associated with the first foreground object following the interaction.
15. The non-transitory computer-readable medium of claim 9 , wherein determining the probability of the observed interaction between the first foreground object and the second foreground object, comprises determining a probability of observing a first segment associated with the first foreground object prior to the interaction and observing a second segment associated with the second foreground object prior to the interaction.
16. The non-transitory computer-readable medium of claim 9 , wherein determining the probability of the observed interaction between the first foreground object and the second foreground object, comprises determining a probability of observing a first segment associated with the first foreground object following the interaction and observing a second segment associated with the second foreground object following the interaction.
17. A video surveillance system, comprising:
a video input source configured to provide an input stream of video frames captured by a video camera, each depicting a scene;
a processor; and
a memory containing a program, which, when executed on the processor is configured to perform an operation for analyzing the scene depicted in the input stream of video frames, the operation comprising:
retrieving a first sequence and a second sequence, each providing an retrieving a first sequence and a second sequence, each providing an ordered string of labels, wherein each label corresponds to a cluster in an adaptive resonance theory (ART) network, wherein the strings of labels are generated by mapping kinematic data vectors generated for a first foreground object and a second foreground object detected in the input stream of video frames, respectively, to nodes of a self-organizing map (SOM) and clustering the nodes of the SOM using the ART network, and wherein the first sequence and the second sequence correspond to an observed interaction between the first foreground object and the second foreground object,
identifying one or more segments in each of the first and second sequences, wherein each segment includes a subsequence of the ordered string of labels in the first and second sequences,
determining a probability of an observed interaction between the first foreground object and the second foreground object, relative to a probability distribution generated from an ngram trie, wherein the ngram trie is generated from a plurality of previously observed sequences, each storing an ordered string of labels assigned to clusters in the ART network for objects detected in the input stream of video frames, and
upon determining the probability the observed interaction between the first foreground object and the second foreground object falls below a specified threshold, issuing an alert to a user of the video surveillance system.
18. The system of claim 17 , wherein identifying one or more segments in the each of the first and second sequences, comprises:
evaluating the first sequence and second sequence, respectively, using a sliding window of length (L), wherein an internal entropy expert and a boundary entropy expert each vote a position in each sliding window subsequence at which to split the sliding window sequence; and
inducing a segment boundary at each position in the first sequence and the second sequence that receives at least a number of votes satisfying a vote count threshold (V).
19. The system of claim 18 , wherein:
the internal entropy expert votes to minimize internal entropy by selecting a position which minimizes internal entropy in each respective sliding window sequence, based on the determined entropies; and
the boundary entropy expert votes by selecting a position in each respective sliding window that maximizes entropy between segments in the sliding window subsequence.
20. The system of claim 18 , further comprising, determining the sliding window length (L) as
L
=
⌈
arg
max
(
H
^
I
)
+
arg
max
(
H
^
B
)
2
⌉
,
wherein Ĥ I and Ĥ B , represent a standardized entropy value for the entropy measure determined for each node in the ngram trie and argmax( ) returns a level of the node in the trie with the greatest entropy values for Ĥ I and Ĥ B .
21. The computer-readable storage medium of claim 18 , wherein the vote count threshold (v) is determined as follows
V
=
⌈
∑
i
=
1
n
arg
max
(
H
^
I
)
i
+
arg
max
(
H
^
B
)
i
2
n
⌉
,
wherein Ĥ I and Ĥ B , represent a standardized entropy value for the entropy measure determined for each node in the ngram trie, wherein n represents a total number of branches in the ngram trie and argmax( ) returns a level of the node in the trie with the greatest entropy values for Ĥ I and Ĥ B .
22. The system of claim 17 , wherein determining the probability of the observed interaction between the first foreground object and the second foreground object, comprises determining a probability of observing a first segment associated with the first foreground object prior to the interaction and observing a second segment associated with the first foreground object following the interaction.
23. The system of claim 17 , wherein determining the probability of the observed interaction between the first foreground object and the second foreground object, comprises determining a probability of observing a first segment associated with the first foreground object prior to the interaction and observing a second segment associated with the second foreground object prior to the interaction.
24. The system of claim 17 , wherein determining the probability of the observed interaction between the first foreground object and the second foreground object, comprises determining a probability of observing a first segment associated with the first foreground object following the interaction and observing a second segment associated with the second foreground object following the interaction.