Anomaly detection system for video surveillance

1 . A system for video anomaly detection for surveillance, comprising:

a memory configured to store a set of exemplars that are feature vectors defining a combination of appearance, size, location, and motion features of objects occurring in normal videos of a scene, wherein each of the exemplars is separated from its closest exemplar by a minimum distance, wherein a feature vector of each of the exemplars includes features indicative of the appearance of an object in the scene, features indicative of the size of the object, features indicative of the location of the object and features indicative of a trajectory of the object tracked in a set of frames of the normal videos, wherein the feature vectors are determined by an object detection neural network and an object tracker; and

a processor configured to:

extract, from a set of input frames of an input video of the scene, one or more input features, the one or more input features comprising input appearance features indicative of the appearance of the object in a frame, input size features indicative of the size of the object in the scene, input location features indicative of the location of the object in the scene, and input trajectory features indicative of a trajectory of the object tracked in the set of input frames of the input video, wherein extracting the one or more input features comprise detecting the object in an input frame from the set of input frames to place a bounding box enclosing the object, wherein the bounding box is generated by the object detection neural network;

combine the input appearance features, the input size features, the input location features, and the input trajectory features to produce an input feature vector;

compare, by an anomaly detector, the input feature vector with each of the exemplars to determine the smallest distance from the input feature vector to its closest exemplar;

declare an anomaly based on the smallest distance being greater than a threshold; and

trigger an alarm in response to the declared anomaly.

2 . The system of claim 1 , wherein the processor is configured to:

track the object in other input frames from the set of input frames to place bounding boxes enclosing the object in multiple input frames;

extract the input appearance features from the bounding box enclosing the object in the input frame;

extract the input size features from the bounding box enclosing the object in the input frame;

extract the input location features from the bounding box enclosing the object in the input frame; and

extract the input trajectory features from the bounding boxes enclosing the object in multiple input frames.

3 . The system of claim 2 , wherein the processor is configured to detect the object and determine the trajectory of the object by executing one or multiple neural networks trained with machine learning.

4 . The system of claim 2 , wherein, to extract the input appearance features from a bounding box enclosing the object in the input frame, the processor is configured to;

extract features of the bounding box enclosing the object in the input frame with a deep neural network trained to classify the object; and

form the input appearance features as an output of an internal layer of the deep neural network.

5 . The system of claim 2 , wherein, to extract the trajectory features from the multiple bounding boxes, the processor is configured to:

determine coordinates of pixels at the center of each of the multiple bounding boxes to form a sequence of coordinates; and

form the trajectory features as horizontal and vertical displacements of the coordinates from one frame to a next frame in the sequence of coordinates.

6 . The system of claim 1 , wherein the processor is configured toto process the input video with a sliding temporal window method to produce the set of input frames at different iterations.

7 . The system of claim 6 , wherein a shift of the sliding window method is fixed.

8 . The system of claim 1 , wherein to determine the closest distance, the processor is configured to determine a distance between the input feature vector and an exemplar as a normalized combination of distances between each group of corresponding features of the input feature vector and the exemplar.

9 . The system of claim 1 , wherein the processor is configured to detect multiple objects in the set of input frames and determine the anomaly by analyzing input exemplars determined for all of the detected objects.

10 . The system of claim 1 , wherein the processor, to determine the exemplars, is configured to:

extract, from frames of the normal video, multiple normal exemplars including combinations of features indicative of the appearance and the trajectory of objects detected in the normal video; and

prune the multiple normal exemplars to produce the exemplars.

11 . A method for video anomaly detection for surveillance, wherein the method uses a processor coupled with stored instructions implementing the method, wherein the instructions, when executed by the processor carry out steps of the method, comprising:

accessing a set of exemplars that are feature vectors defining a combination of appearance, size, location, and motion features of objects occurring in normal videos of a scene, wherein each of the exemplars is separated from its closest exemplar by a minimum distance, wherein a feature vector of each of the exemplars includes features indicative of the appearance of an object in the scene, features indicative of the size of the object, features indicative of the location of the object and features indicative of a trajectory of the object tracked in a set of frames of the normal videos, wherein the feature vectors are determined by an object detection neural network and an object tracker;

extracting, from a set of input frames of an input video of the scene, one or more input features, the one or more input features comprising input appearance features indicative of the appearance of the object in a frame, input size features indicative of the size of the object in the scene, input location features indicative of the location of the object in the scene, and input trajectory features indicative of a trajectory of the object tracked in the set of input frames of the input video, wherein extracting the one or more input features comprise detecting the object in an input frame from the set of input frames to place a bounding box enclosing the object, wherein the bounding box is generated by the object detection neural network;

combining the input appearance features, the input size features, the input location features, and the input trajectory features to produce an input feature vector;

comparing the input feature vector with each of the exemplars to determine the smallest distance from the input feature vector to its closest exemplar;

declaring an anomaly when the smallest distance is greater than a threshold; and

triggering an alarm in response to the anomaly.

12 . The method of claim 11 , further comprising:

tracking the object in other input frames from the set of input frames to place bounding boxes enclosing the object in multiple input frames;

extracting the input appearance features from the bounding box enclosing the object in the input frame;

extracting the input size features from the bounding box enclosing the object in the input frame;

extracting the input location features from the bounding box enclosing the object in the input frame; and

extracting the input trajectory features from the bounding boxes enclosing the object in multiple input frames.

13 . The method of claim 12 , further comprising:

executing one or multiple neural networks trained with machine learning to detect the object and determine the trajectory of the object.

14 . The method of claim 12 , wherein, to extract the input appearance features from a bounding box defined by a bounding box enclosing the object in the input frame, the method comprises:

extracting features of the bounding box enclosing the object in the input frame with a deep neural network trained to classify the object; and

forming the input appearance features as an output of an internal layer of the deep neural network.

15 . The method of claim 12 , wherein, to extract the trajectory features from the multiple bounding boxes, the method comprises:

determining coordinates of pixels at the center of each of the multiple bounding boxes to form a sequence of coordinates; and

forming the trajectory features as horizontal and vertical displacements of the coordinates from one frame to next frame in the sequence of coordinates.

16 . A non-transitory computer-readable storage medium embodied thereon a program executable by a processor for performing a method for video anomaly detection for surveillance, the method comprising:

accessing a set of exemplars that are feature vectors defining a combination of appearance, size, location, and motion features of objects occurring in normal videos of a scene, wherein each of the exemplars is separated from its closest exemplar by a minimum distance, wherein a feature vector of each of the exemplars includes features indicative of the appearance of an object in the scene, features indicative of the size of the object, features indicative of the location of the object and features indicative of a trajectory of the object tracked in a set of frames of the normal videos, wherein the feature vectors are determined by an object detection neural network and an object tracker;

extracting, from a set of input frames of an input video of the scene, one or more input features, the one or more input features comprising input appearance features indicative of the appearance of the object in a frame, input size features indicative of the size of the object in the scene, input location features indicative of the location of the object in the scene, and input trajectory features indicative of a trajectory of the object tracked in the set of input frames of the input video, wherein extracting the one or more input features comprise detecting the object in an input frame from the set of input frames to place a bounding box enclosing the object, wherein the bounding box is generated by the object detection neural network;

combining the input appearance features, the input size features, the input location features, and the input trajectory features to produce an input feature vector;

comparing, by an anomaly detector, the input feature vector with each of the exemplars to determine the smallest distance from the input feature vector to its closest exemplar;

declaring an anomaly when the smallest distance is greater than a threshold; and

triggering an alarm in response to the anomaly.