Anomalous stationary object detection and reporting
View Patent ↗Techniques are disclosed for analyzing a scene depicted in an input stream of video frames captured by a video camera. The techniques include receiving data for an object within the scene and determining whether the object has remained substantially stationary within the scene for at least a threshold period. If the object is determined to have remained stationary for at least the threshold period, a rareness score is calculated for the object to indicate a likelihood of the object being stationary to an observed degree at an observed location. The rareness score may use a learning model to take into account previous stationary and/or non-stationary behavior of objects within the scene. In general, the learning model may be updated based on observed stationary and/or non-stationary behaviors of the objects. If the rareness score meets reporting conditions, the stationary object event may be reported.
1. A computer-implemented method for analyzing a scene depicted in an input stream of video frames captured by a video camera, the method comprising:
receiving, from the video camera, the input stream of video frames;
identifying an object depicted in the input stream of video frames received from the video camera;
determining, via one or more processors, whether the object has remained at the observed location in the scene for at least a threshold period;
determining whether the object has remained substantially stationary during the threshold period;
upon determining the object has remained substantially stationary and remained within the scene for at least the threshold period, determining a rareness score for the object, wherein the rareness score indicates a likelihood of the object being substantially stationary at the observed location, and wherein the rareness score is determined based at least on automatically learned stationary and non-stationary count maps indicating observed frequencies of stationary and non-stationary objects in regions of the scene, respectively, and wherein the rareness score has form
R
=
S
×
(
1.0
-
F
)
(
1
-
std
dev
average
)
,
where S represents a stationarity score given by S=N i /N s , N i being a number of intersection pixels of an object's bounding box(es) during a period in which the object has been tracked in the scene and N s being a union of pixels covered by the bounding box(es) during the tracking period, and F represents a frequency score given by F=N s /N a , N s being a sum of counts of stationary objects over tracking pixels, and N a being a sum of stationary and non-stationary counts of all objects over all pixels; and
upon determining the rareness score exceeds a threshold value, generating an alert.
2. The method of claim 1 , wherein determining whether the object has remained substantially stationary includes determining whether the bounding box of the object in each of a given number of the input stream of video frames overlap for at least the threshold period.
3. The method of claim 1 , further comprising:
storing a list of substantially stationary objects no longer visible in the scene;
upon determining the object appears from behind an other object, comparing the object to the substantially stationary objects in the list; and
upon determining the object matches one of the substantially stationary objects in the stored list, adding time the object is substantially stationary before leaving the scene, time between the object leaving the scene and appearing from behind the other object, and time during which the object is substantially stationary after appearing from behind the other object, to obtain the time the object has remained at the observed location in the scene.
4. The method of claim 1 , further comprising:
storing a list of objects previously associated with rareness scores that exceeded the threshold value; and
if the object is determined to have remained stationary and remained within the scene for at least the threshold period, comparing pixel locations of the object with pixel locations of objects in the stored list,
wherein the alert is generated or the object is reported upon determining pixel locations of the object do not match the pixel locations of any of the objects in the stored list.
5. The method of claim 1 , further comprising, updating one or both of the stationary and non-stationary count maps based on the observed stationary and/or non-stationary behavior of the object.
6. The method of claim 5 , wherein the updating includes updating pixel count value(s) in area(s) covered by the object in the stationary count map or the non-stationary count map if the object remained stationary or non-stationary, respectively.
7. A non-transitory computer-readable storage medium storing instructions, which when executed by a computer system, perform operations for analyzing a scene depicted in an input stream of video frames captured by a video camera, the operations comprising:
receiving, from the video camera, the input stream of video frames;
identifying an object depicted in the input stream of video frames received from the video camera;
determining, via one or more processors, whether the object has remained at the observed location in the scene for at least a threshold period;
determining whether the object has remained substantially stationary during the threshold period;
upon determining the object has remained substantially stationary and remained within the scene for at least the threshold period, determining a rareness score for the object, wherein the rareness score indicates a likelihood of the object being substantially stationary at the observed location, wherein the rareness score is determined based at least on automatically learned stationary and non-stationary count maps indicating observed frequencies of stationary and non-stationary objects in regions of the scene, respectively, and wherein the rareness score has form
R
=
S
×
(
1.0
-
F
)
(
1
-
std
dev
average
)
,
where S represents a stationarity score given by S=N i /N s , N i being a number of intersection pixels of an object's bounding box(es) during a period in which the object has been tracked in the scene and N s being a union of pixels covered by the bounding box(es) during the tracking period, and F represents a frequency score given by F=N s /N a , N s being a sum of counts of stationary objects over tracking pixels, and N a being a sum of stationary and non-stationary counts of all objects over all pixels; and
upon determining the rareness score exceeds a threshold value, generating an alert.
8. The computer-readable storage medium of claim 7 , wherein determining whether the object has remained substantially stationary includes determining whether the bounding box of the object in each of a given number of the input stream of video frames overlap for at least the threshold period.
9. The computer-readable storage medium of claim 7 , the operations further comprising:
storing a list of substantially stationary objects no longer visible in the scene;
upon determining the object appears from behind an other object, comparing the object to the substantially stationary objects in the list; and
upon determining the object matches one of the substantially stationary objects in the stored list, adding time the object is substantially stationary before leaving the scene, time between the object leaving the scene and appearing from behind the other object, and time during which the object is substantially stationary after appearing from behind the other object, to obtain the time the object has remained at the observed location in the scene.
10. The computer-readable storage medium of claim 7 , the operations further comprising:
storing a list of objects previously associated with rareness scores that exceeded the threshold value; and
if the object is determined to have remained stationary and remained within the scene for at least the threshold period, comparing pixel locations of the object with pixel locations of objects in the stored list,
wherein the alert is generated or the object is reported upon determining pixel locations of the object do not match the pixel locations of any of the objects in the stored list.
11. The computer-readable storage medium of claim 7 , the operations further comprising, updating one or both of the stationary and non-stationary count maps based on the observed stationary and/or non-stationary behavior of the object.
12. The computer-readable storage medium of claim 11 , wherein the updating includes updating pixel count value(s) in area(s) covered by the object in the stationary count map or the non-stationary count map if the object remained stationary or non-stationary, respectively.
13. A system, comprising:
a video camera;
a processor; and
a memory, wherein the memory includes an application program configured to perform operations for analyzing a scene depicted in an input stream of video frames captured by the video camera, the operations comprising:
receiving, from the video camera, the input stream of video frames,
identifying an object depicted in the input stream of video frames received from the video camera,
determining, via the application execution the processor, whether the object has remained at the observed location in the scene for at least a threshold period,
determining whether the object has remained substantially stationary during the threshold period,
upon determining the object has remained substantially stationary and remained within the scene for at least the threshold period, determining a rareness score for the object, wherein the rareness score indicates a likelihood of the object being substantially stationary at the observed location, wherein the rareness score is determined based at least on automatically learned stationary and non-stationary count maps indicating observed frequencies of stationary and non-stationary objects in regions of the scene, respectively, and wherein the rareness score has form
R
=
S
×
(
1.0
-
F
)
(
1
-
std
dev
average
)
,
where S represents a stationarity score given by S=Ni/Ns, Ni being a number of intersection pixels of the object's bounding box(es) during a period in which the objects tracked and Ns being a union of pixels covered by the bounding box(es) during the tracking period, and F represents a frequency score given by F=Ns/Na, Ns being a sum of counts of stationary objects over tracking pixels, and Na being a sum of stationary and non-stationary counts of all objects over all pixels, and
upon determining the rareness score exceeds a threshold value, generating an alert.
14. The system of claim 13 , wherein determining whether the object has remained substantially stationary includes determining whether the bounding box of the object in each of a given number of the input stream of video frames overlap for at least the threshold period.
15. The system of claim 13 , the operations further comprising:
storing a list of substantially stationary objects no longer visible in the scene;
upon determining the object appears from behind an other object, comparing the object to the substantially stationary objects in the list; and
upon determining the object matches one of the substantially stationary objects in the stored list, adding time the object is substantially stationary before leaving the scene, time between the object leaving the scene and appearing from behind the other object, and time during which the object is substantially stationary after appearing from behind the other object, to obtain the time the object has remained at the observed location in the scene.
16. The system of claim 13 , the operations further comprising, updating one or both of the stationary and non-stationary count maps based on the observed stationary and/or non-stationary behavior of the object.
17. The system of claim 16 , wherein the updating includes updating pixel count value(s) in area(s) covered by the object in the stationary count map or the non-stationary count map if the object remained stationary or non-stationary, respectively.