Unsupervised learning of temporal anomalies for a video surveillance system
View Patent ↗Techniques are described for analyzing a stream of video frames to identify temporal anomalies. A video surveillance system configured to identify when agents depicted in the video stream engage in anomalous behavior, relative to the time-of-day (TOD) or day-of-week (DOW) at which the behavior occurs. A machine-learning engine may establish the normalcy of a scene by observing the scene over a specified period of time. Once the observations of the scene have matured, the actions of agents in the scene may be evaluated and classified as normal or abnormal temporal behavior, relative to the past observations.
1. A computer-implemented method for analyzing a sequence of video frames depicting a scene captured by a video camera, the method comprising:
receiving a set of kinematic data derived by a computer vision engine observing a foreground object in one of the frames of video;
receiving temporal data specifying when the foreground object was observed in one of the frames of video;
passing the set of kinematic data and the temporal data to an adaptive resonance theory (ART) network, wherein the ART network models observed behavior of a plurality of foreground objects observed in the scene, relative to the kinematic data and the temporal data;
evaluating one or more clusters of the ART network to determine whether the set of kinematic data and temporal data passed to the ART network are indicative of an occurrence of a temporally anomaly; and
upon determining a temporal anomaly has occurred, publishing an alert message.
2. The computer-implemented method of claim 1 , wherein the temporal data specifies a time of day when the foreground object was observed in one of the frames of video.
3. The computer-implemented method of claim 2 , further comprising, prior to publishing the alert message, determining that the ART network has modeled the observed behavior of the plurality of foreground objects in the scene for a threshold period of time.
4. The computer-implemented method of claim 1 , wherein the temporal data specifies a day of week when the foreground object was observed in one of the frames of video.
5. The computer-implemented method of claim 4 , further comprising, prior to publishing the alert message, determining that the ART network has modeled the observed behavior of the plurality of foreground objects in the scene for a threshold period of time.
6. The computer-implemented method of claim 1 , wherein the kinematic data includes a spatial position in a frame of video where the foreground object has been observed by the computer vision engine, a pixel-height of the foreground object and a pixel width of the foreground object.
7. The computer-implemented method of claim 1 , wherein the kinematic data includes a spatial position in a frame of video where the foreground object has been observed by the computer vision engine, a velocity in a horizontal dimension of the foreground object, and a velocity of the foreground object in a vertical dimension.
8. The computer-implemented method of claim 1 , wherein the kinematic data includes a spatial position in one of the frames of video where the foreground object has been observed by the computer vision engine, a rate of acceleration of the foreground object in a horizontal dimension and a rate of acceleration of the foreground object in a vertical dimension.
9. The computer-implemented method of claim 1 , further comprising, merging two or more overlapping clusters in the ART network.
10. The computer-implemented method of claim 1 , further comprising, removing one or more clusters in the ART network that have not been updated for a period exceeding a specified time period.
11. The computer-implemented method of claim 1 , further comprising: updating the ART network based on the kinematic data and the temporal data.
12. The computer-implemented method of claim 11 , wherein updating the ART network comprises one of:
(i) generating a new cluster at an initial position determined form the passed data inputs, wherein the new cluster includes an initial mean and a variance, and wherein the new cluster is bounded by a specified distance from the initial position for each dimension of data passed to the input layer;
(ii) updating a previously generated cluster by updating the position, mean and variance of the previously generated cluster.
13. The computer-implemented method of claim 1 , wherein evaluating the one or more clusters of the ART network to determine whether the set of kinematic data and temporal data passed to the ART network are indicative of the occurrence of a temporally anomaly comprises one of (i) determining that the ART network generated a new cluster to model the kinematic data and the temporal data in the ART network and (ii) determining that the ART network mapped the kinematic data and the temporal data to a cluster in the ART network that has not been reinforced with a threshold number of inputs.
14. A computer-readable storage medium containing a program which, when executed by a processor, performs an operation for analyzing a sequence of video frames depicting a scene captured by a video camera, the operation comprising:
receiving a set of kinematic data derived by a computer vision engine observing a foreground object in one of the frames of video;
receiving temporal data specifying when the foreground object was observed in one of the frames of video;
passing the set of kinematic data and the temporal data to an adaptive resonance theory (ART) network, wherein the ART network models observed behavior of a plurality of foreground objects observed in the scene, relative to the kinematic data and the temporal data;
evaluating one or more clusters of the ART network to determine whether the set of kinematic data and temporal data passed to the ART network are indicative of an occurrence of a temporally anomaly; and
upon determining a temporal anomaly has occurred, publishing an alert message.
15. The computer-readable storage medium of claim 14 , wherein the temporal data specifies at least one of a time of day when the foreground object was observed in one of the frames of video and a day of week when the foreground object was observed in one of the frames of video.
16. The computer-readable storage medium of claim 15 , wherein the operation further comprises, prior to publishing the alert message, determining that the ART network has modeled the observed behavior of the plurality of foreground objects in the scene for a threshold period of time.
17. The computer-readable storage medium of claim 14 , wherein the kinematic data includes at least one of: (i) a spatial position in a frame of video where the foreground object has been observed by the computer vision engine and a pixel-height of the foreground object and a pixel width of the foreground object (ii) a spatial position in one of the frames of video where the foreground object has been observed by the computer vision engine, a velocity in a horizontal dimension of the foreground object, and a velocity of the foreground object in a vertical dimension, and (iii) a spatial position in one of the frames of video where the foreground object has been observed by the computer vision engine, a rate of acceleration of the foreground object in a horizontal dimension and a rate of acceleration of the foreground object in a vertical dimension.
18. The computer-readable storage medium of claim 14 , wherein the operation further comprises: updating the ART network based on the kinematic data and the temporal data.
19. The computer-readable storage medium of claim 18 , wherein updating the ART network comprises one of:
(i) generating a new cluster at an initial position determined form the passed data inputs, wherein the new cluster includes an initial mean and a variance, and wherein the new cluster is bounded by a specified distance from the initial position for each dimension of data passed to the input layer;
(ii) updating a previously generated cluster by updating the position, mean and variance of the previously generated cluster.
20. A system, comprising:
a video input source configured to provide a sequence of video frames, each depicting a scene;
a processor; and
a memory containing a program, which, when executed on the processor is configured to perform an operation for analyzing the scene, as depicted by the sequence of video frames captured by the video input source, the operation comprising:
receiving a set of kinematic data derived by a computer vision engine observing a foreground object in one of the frames of video,
receiving temporal data specifying when the foreground object was observed in one of the frames of video,
passing the set of kinematic data and the temporal data to an adaptive resonance theory (ART) network, wherein the ART network models observed behavior of a plurality of foreground objects observed in the scene, relative to the kinematic data and the temporal data,
evaluating one or more clusters of the ART network to determine whether the set of kinematic data and temporal data passed to the ART network are indicative of an occurrence of a temporally anomaly, and
upon determining a temporal anomaly has occurred, publishing an alert message.
21. The system of claim 20 , wherein the temporal data specifies at least one of a time of day when the foreground object was observed in one of the frames of video and a day of week when the foreground object was observed in one of the frames of video.
22. The system of claim 21 , wherein the operation further comprises, prior to publishing the alert message, determining that the ART network has modeled the observed behavior of the plurality of foreground objects in the scene for a threshold period of time.
23. The system of claim 20 , wherein the kinematic data includes at least one of: (i) a spatial position in a frame of video where the foreground object has been observed by the computer vision engine and a pixel-height of the foreground object and a pixel width of the foreground object (ii) a spatial position in one of the frames of video where the foreground object has been observed by the computer vision engine, a velocity in a horizontal dimension of the foreground object, and a velocity of the foreground object in a vertical dimension, and (iii) a spatial position in one of the frames of video where the foreground object has been observed by the computer vision engine, a rate of acceleration of the foreground object in a horizontal dimension and a rate of acceleration of the foreground object in a vertical dimension.
24. The system of claim 20 , wherein the operation further comprises: updating the ART network based on the kinematic data and the temporal data.
25. The system of claim 24 , wherein updating the ART network comprises one of:
(i) generating a new cluster at an initial position determined form the passed data inputs, wherein the new cluster includes an initial mean and a variance, and wherein the new cluster is bounded by a specified distance from the initial position for each dimension of data passed to the input layer;
(ii) updating a previously generated cluster by updating the position, mean and variance of the previously generated cluster.