Unsupervised learning of feature anomalies for a video surveillance system
View Patent ↗Techniques are disclosed for analyzing a scene depicted in an input stream of video frames captured by a video camera. In one embodiment, e.g., a machine learning engine may include statistical engines for generating topological feature maps based on observations and a detection module for detecting feature anomalies. The statistical engines may include adaptive resonance theory (ART) networks which cluster observed position-feature characteristics. The statistical engines may further reinforce, decay, merge, and remove clusters. The detection module may calculate a rareness value relative to recurring observations and data in the ART networks. Further, the sensitivity of detection may be adjusted according to the relative importance of recently observed anomalies.
1. A computer-implemented method for analyzing a scene, the method comprising:
receiving kinematic and feature data for an object in the scene;
determining, via one or more processors, a position-feature vector from the received data, the position-feature vector representing a location and one or more feature values at the location;
retrieving a feature map corresponding to the position-feature vector, wherein the feature map includes one or more position-feature clusters, wherein the feature map includes one or more adaptive resonance theory (ART) network clusters;
determining a rareness value for the object based at least on the position feature vector and the feature map, wherein the rareness value is determined based on at least a pseudo-Mahalanobis distance of the position-feature vector to a closest one of the clusters and on statistical relevance of clusters less than a threshold distance from the position-feature vector; and
reporting the object as anomalous if the rareness value meets given criteria.
2. The method of claim 1 , further comprising, updating the feature map using the position-feature vector.
3. The method of claim 1 , wherein distances to the clusters less than the threshold distance are determined as mean-squared error distances.
4. The method of claim 3 , wherein the rareness value is determined as
Rareness
=
1
-
[
α
*
Aprior
+
(
1
-
α
)
*
Aposteriori
]
,
where
Aprior
=
∑
statistical
relevance
of
cluster
(
s
)
having
distance
less
than
threshold
total
number
of
observations
,
Aposteriori is the pseudo-Mahalanobis distance of the position-feature vector to the closest cluster, and α is a constant.
5. The method of claim 1 , wherein mean and variance of the clusters are actual mean and variance of input vectors which have mapped to the clusters, wherein each cluster is weighted by statistical relevance, and wherein the clusters are reinforced, decayed, merged, and removed by a statistical engine.
6. The method of claim 1 , further comprising, modifying the feature map based on a frequency of observed anomalies before determining the rareness value.
7. The method of claim 6 , wherein modifying the feature map includes increasing variances of clusters in the feature map based on counts of observed anomalies associated with each cluster.
8. The method claim 1 , wherein the one or more appearance features include at least one of area, area derivative, shadow, shininess, internal energy, area, entropy, gradient histogram, hue, global positioning system (GPS) coordinates, and radio-frequency identification (RFID) tags.
9. The method of claim 1 , wherein the kinematic and feature data are extracted from video frames captured by a video camera.
10. A non-transitory computer-readable storage medium storing instructions, which when executed by a computer system, perform operations for analyzing a scene, the operations comprising:
receiving kinematic and feature data for an object in the scene;
determining, via one or more processors, a position-feature vector from the received data, the position-feature vector representing a location and one or more feature values at the location;
retrieving a feature map corresponding to the position-feature vector, wherein the feature map includes one or more position-feature clusters, wherein the feature map includes one or more adaptive resonance theory (ART) network clusters;
determining a rareness value for the object based at least on the position feature vector and the feature map, wherein the rareness value is determined based on at least a pseudo-Mahalanobis distance of the position-feature vector to a closest one of the clusters and on statistical relevance of clusters less than a threshold distance from the position-feature vector; and
reporting the object as anomalous if the rareness value meets given criteria.
11. The computer-readable storage medium of claim 10 , the operations further comprising, updating the feature map using the position-feature vector.
12. The computer-readable storage medium of claim 10 , wherein distances to the clusters less than the threshold distance are determined as mean-squared error distances.
13. The computer-readable storage medium of claim 12 , wherein the rareness value is determined as
Rareness
=
1
-
[
α
*
Aprior
+
(
1
-
α
)
*
Aposteriori
]
,
where
Aprior
=
∑
statistical
relevance
of
cluster
(
s
)
having
distance
less
than
threshold
total
number
of
observations
,
Aposteriori is the pseudo-Mahalanobis distance of the position-feature vector to the closest cluster, and α is a constant.
14. The computer-readable storage medium of claim 10 , wherein mean and variance of the clusters are actual mean and variance of input vectors which have mapped to the clusters, wherein each cluster is weighted by statistical relevance, and wherein the clusters are reinforced, decayed, merged, and removed by a statistical engine.
15. The computer-readable storage medium of claim 10 , the operations further comprising, modifying the feature map based on a frequency of observed anomalies before determining the rareness value.
16. The computer-readable storage medium of claim 15 , wherein modifying the feature map includes increasing variances of clusters in the feature map based on counts of observed anomalies associated with each cluster.
17. The computer-readable storage medium of claim 10 , wherein the one or more appearance features include at least one of area, area derivative, shadow, shininess, internal energy, area, entropy, gradient histogram, hue, global positioning system (GPS) coordinates, and radio-frequency identification (RFID) tags.
18. The computer-readable storage medium of claim 17 , wherein the kinematic and feature data are extracted from video frames captured by a video camera.
19. A system, comprising:
a processor; and
a memory, wherein the memory includes an application program configured to perform operations for analyzing a scene, the operations comprising:
receiving kinematic and feature data for an object in the scene,
determining, via one or more processors, a position-feature vector from the received data, the position-feature vector representing a location and one or more feature values at the location,
retrieving a feature map corresponding to the position-feature vector, wherein the feature map includes one or more position-feature clusters, wherein the feature map includes one or more adaptive resonance theory (ART) network clusters,
determining a rareness value for the object based at least on the position feature vector and the feature map, wherein the rareness value is determined based on at least a pseudo-Mahalanobis distance of the position-feature vector to a closest one of the clusters and on statistical relevance of clusters less than a threshold distance from the position-feature vector, and
reporting the object as anomalous if the rareness value meets given criteria.
20. The system of claim 19 , wherein distances to the clusters less than the threshold distance are determined as mean-squared error distances.
21. The system of claim 20 , wherein the rareness value is determined as
Rareness
=
1
-
[
α
*
Aprior
+
(
1
-
α
)
*
Aposteriori
]
,
where
Aprior
=
∑
statistical
relevance
of
cluster
(
s
)
having
distance
less
than
threshold
total
number
of
observations
,
Aposteriori is the pseudo-Mahalanobis distance of the position-feature vector to the closest cluster, and α is a constant.
22. The system of claim 19 , wherein mean and variance of the clusters are actual mean and variance of input vectors which have mapped to the clusters, wherein each cluster is weighted by statistical relevance, and wherein the clusters are reinforced, decayed, merged, and removed by a statistical engine.