INTELLIGENT VIDEO SURVEILLANCE SYSTEM AND METHOD
An intelligent video surveillance system is disclosed which performs real-time analytics on a live video stream. The system includes a training database populated with frames of actual video of objects of interest taken in a relevant environment. A subset of the frames include bounding boxes and/or bounding polygons which can be augmented. The training database also includes classification/annotation of data/labels relevant to the object of interest, a person carrying the object of interest, and/or the background or environment. The training database is searchable by the classification/annotation of data/labels.
1 . A method comprising:
receiving a first video stream, the first video stream comprising a first one or more classifications;
selecting a first plurality of frames from the first video stream;
detecting a presence of a first object in one or more frames of the first plurality of frames;
generating annotated frames by:
inserting a bounding box in an area of the first object in the one or more frames, and
annotating the one or more frames with the first one or more classifications;
storing the annotated frames in a database, the database configured to be searchable by at least one classification of the first one or more classifications;
training one or more detection models using the annotated frames, the training comprising varying one or more parameters of the respective one or more detection models;
receiving a second video stream, the second video stream comprising a second one or more classifications;
automatically selecting, based on a determination that the second one or more classifications are similar to the first one or more classifications, a first detection model of the one or more detection models;
analyzing a second plurality of frames from the second video stream using the first detection model to detect a presence of a second object;
determining a ratio of true positive events to false positive events (“TP/FP”) identified by the first detection model;
determining a ratio of false positive events to true positive events (“FP/TP”) identified by the first detection model;
converting the FP/TP to a percentage (“% FP/TP);
generating an evaluation of a performance of the first detection model based on one or more metrics, the one or more metrics comprising a ratio of the % FP/TP to the TP/FP; and
automatically selecting, based on the evaluation, a second detection model of the one or more detection models to replace the first detection model.
2 . The method of claim 1 , wherein one or more of the first video stream and the second video stream comprise footage from a security camera in a real-life environment.
3 . The method of claim 1 , wherein the first video stream comprises footage from an artificially-created environment that simulates a real scenario.
4 . The method of claim 1 , wherein one or more of the first object and the second object are directly visible.
5 . The method of claim 1 , wherein one or more of the first object and the second object are obscured from being directly visible.
6 . The method of claim 1 , wherein the bounding box comprise a polygon that circumscribes at least a portion of the first object.
7 . The method of claim 1 , wherein the one or more first classifications and the one or more second classifications comprise one or more of a type of object, an orientation of object, color, lighting, clarity, contrast, source camera information.
8 . The method of claim 1 , wherein the one or more first classifications and the one or more second classifications comprise a hierarchal order.
9 . The method of claim 1 , further comprising:
retraining the first model based on the evaluation using the second one or more classifications.
10 . The method of claim 1 , wherein one or more of the inserting and the annotating is done manually by a user.
11 . The method of claim 1 , further comprising:
adjusting one or more of a location and a size of one or more of the bounding boxes.
12 . The method of claim 1 , wherein the one or more parameters comprise one or more of a number of labels, a number of images, a number of frames, a number of iterations, a max iteration value, a test iteration value, a test interval, a momentum value, a ratio value, a learning rate, a weight decay, confidence score, event duration, pixel area size, object speed, minimum range of object movement, average object size, and average pixel speed.
13 . The method of claim 1 , wherein the one or more metrics further comprise a number of TP events, a number of FP events, a score value, an average score value, a label performance value, a score by distance value.
14 . A system comprising:
a processor operatively coupled to a memory configured to store computer-readable instructions that, when executed by the processor, cause the processor to:
receive a first video stream, the first video stream comprising a first one or more classifications;
select a first plurality of frames from the first video stream;
detect a presence of a first object in one or more frames of the first plurality of frames;
generate annotated frames by:
inserting a bounding box in an area of the first object in the one or more frames, and
annotating the one or more frames with the first one or more classifications;
store the annotated frames in a database, the database configured to be searchable by at least one classification of the one or more classifications;
train one or more detection models using the annotated frames, the training comprising varying one or more parameters of the respective one or more detection models;
receive a second video stream, the second video stream comprising a second one or more classifications;
automatically select, based on a determination that the second one or more classifications are similar to the first one or more classifications, a first detection model of the one or more detection models;
analyze a second plurality of frames from the second video stream using the first detection model of the one or more detection models to detect a presence of a second object;
determine a ratio of true positive events to false positive events (“TP/FP”) identified by the first detection model;
determine a ratio of false positive events to true positive events (“FP/TP”) identified by the first detection model;
convert the FP/TP to a percentage (“% FP/TP);
generate an evaluation of a performance of the first detection model based on one or more metrics, the one or more metrics comprising a ratio of the % FP/TP to the TP/FP; and
automatically select, based on the evaluation, a second detection model of the one or more detection models to replace the first detection model.
15 . The system of claim 14 , wherein one or more of the first video stream and the second video stream comprise footage from a security camera in a real-life environment.
16 . The system of claim 14 , wherein the first video stream comprises footage from an artificially-created environment that simulates a real scenario.
17 . The system of claim 14 , wherein one or more of the first object and the second object are directly visible.
18 . The system of claim 14 , wherein one or more of the first object and the second object are obscured from being directly visible.
19 . The system of claim 14 , wherein the bounding box comprise a polygon that circumscribes at least a portion of the first object.
20 . The system of claim 14 , wherein the one or more first classifications and the one or more second classifications comprise one or more of a type of object, an orientation of object, color, lighting, clarity, contrast, source camera information.
21 . The system of claim 20 , wherein the one or more first classifications and the one or more second classifications comprise a hierarchal order.
22 . The system of claim 14 , wherein the computer-readable instructions, when executed, further cause the processor to:
retrain the first model based on the evaluation using the second one or more classifications.
23 . The system of claim 14 , wherein one or more of the inserting and the annotating is done manually by a user.
24 . The system of claim 14 , wherein the computer-readable instructions, when executed, further cause the processor to:
adjust one or more of a location and a size of one or more of the bounding boxes.
25 . The system of claim 14 , wherein the one or more parameters comprise one or more of a number of labels, a number of images, a number of frames, a number of iterations, a max iteration value, a test iteration value, a test interval, a momentum value, a ratio value, a learning rate, a weight decay, confidence score, event duration, pixel area size, object speed, minimum range of object movement, average object size, and average pixel speed.
26 . The system of claim 14 , wherein the one or more metrics further comprise a number of TP events, a number of FP events, a score value, an average score value, a label performance value, a score by distance value.