Inter-trajectory anomaly detection using adaptive voting experts in a video surveillance system
A sequence layer in a machine-learning engine configured to learn from the observations of a computer vision engine. In one embodiment, the machine-learning engine uses the voting experts to segment adaptive resonance theory (ART) network label sequences for different objects observed in a scene. The sequence layer may be configured to observe the ART label sequences and incrementally build, update, and trim, and reorganize an ngram trie for those label sequences. The sequence layer computes the entropies for the nodes in the ngram trie and determines a sliding window length and vote count parameters. Once determined, the sequence layer may segment newly observed sequences to estimate the primitive events observed in the scene as well as issue alerts for inter-sequence and intra-sequence anomalies.
1. A computer-implemented method for evaluating objects detected in a video stream, the method comprising:
detecting a plurality of foreground objects present in the video stream;
for each foreground object of the plurality of foreground objects, building a trajectory characterizing each foreground object in a series of successive frames of the single video stream, the trajectory comprising a vector comprising kinematic data for the foreground object from when the foreground object is first observed in the single video stream to when the foreground object leaves the single video stream;
storing each trajectory in a memory;
identifying one or more patterns of behavior of objects in the video stream using the stored trajectories;
detecting a successive foreground object in the video stream;
building a trajectory of the successive foreground object;
determining a probability distribution that the trajectory of the successive object is anomalous based on the stored trajectories; and
if the trajectory of the successive object is determined to be anomalous, generating an alert.
2. The method of claim 1 , wherein the sequences of vectors are mapped to nodes of a self organizing map (SOM) and wherein nodes of the SOM are clustered using an adaptive resonance theory (ART) network to generate a sequence of SOM nodes.
3. The method of claim 2 , wherein determining the anomalous interaction comprises determining a probability of observing the sequence of SOM nodes of the successive object in conjunction with a probability of observing a sequence of SOM nodes associated with the plurality of foreground objects.
4. The method of claim 1 , wherein each of the plurality of foreground objects are classified as being instances of an object type sharing similar microfeature vectors characterizing the foreground object in the series of successive frames of the video stream.
5. A non-transitory computer storage medium, which, when executed on a processor, performs an operation for evaluating objects detected in a video stream, comprising:
detecting a plurality of foreground objects present in the video stream;
for each foreground object of the plurality of foreground objects, building a trajectory characterizing each foreground object in a series of successive frames of the single video stream, the trajectory comprising a vector comprising kinematic data for the foreground object from when the foreground object is first observed in the single video stream to when the foreground object leaves the single video stream;
storing each trajectory in a memory;
identifying one or more patterns of behavior of objects in the video stream using the stored trajectories;
detecting a successive foreground object in the video stream;
building a trajectory of the successive foreground object;
determining a probability distribution that the trajectory of the successive object is anomalous based on the stored trajectories; and
if the trajectory of the successive object is determined to be anomalous, generating an alert.
6. The non-transitory computer storage medium of claim 5 , wherein the sequences of vectors are mapped to nodes of a self organizing map (SOM) and wherein nodes of the SOM are clustered using an adaptive resonance theory (ART) network to generate a sequence of SOM nodes.
7. The non-transitory computer storage medium of claim 6 , wherein determining the anomalous interaction comprises determining a probability of observing the sequence of SOM nodes of the successive object in conjunction with a probability of observing a sequence of SOM nodes associated with the plurality of foreground objects.
8. The non-transitory computer storage medium of claim 7 , wherein each of the plurality of foreground objects are classified as being instances of an object type sharing similar microfeature vectors characterizing the foreground object in the series of successive frames of the video stream.
9. A video surveillance system, comprising:
a video source configured to provide a single input video stream captured by a video camera;
a processor; and
a memory containing a program, which, when executed on the processor is configured to perform an operation for evaluating objects detected in the single input video stream, the operation comprising:
detecting a plurality of foreground objects present in the video stream;
for each foreground object of the plurality of foreground objects, building a trajectory characterizing each foreground object in a series of successive frames of the single video stream, the trajectory comprising a vector comprising kinematic data for the foreground object from when the foreground object is first observed in the single video stream to when the foreground object leaves the single video stream;
storing each trajectory in the memory;
identifying one or more patterns of behavior of objects in the video stream using the stored trajectories;
detecting a successive foreground object in the video stream;
building a trajectory of the successive foreground object;
determining a probability distribution that the trajectory of the successive object is anomalous based on the stored trajectories; and
if the trajectory of the successive object is determined to be anomalous, generating an alert.
10. The system of claim 9 , wherein the sequences of vectors are mapped to nodes of a self organizing map (SOM) and wherein nodes of the SOM are clustered using an adaptive resonance theory (ART) network to generate a sequence of SOM nodes.
11. The system of claim 10 , wherein determining the anomalous interaction comprises determining a probability of observing the sequence of SOM nodes of the successive object in conjunction with a probability of observing a sequence of SOM nodes associated with the plurality of foreground objects.
12. The system of claim 11 , wherein each of the plurality of foreground objects are classified as being instances of an object type sharing similar microfeature vectors characterizing the foreground object in the series of successive frames of the video stream.