Background model for complex and dynamic scenes
View Patent ↗Techniques are disclosed for learning and modeling a background for a complex and/or dynamic scene over a period of observations without supervision. A background/foreground component of a computer vision engine may be configured to model a scene using an array of ART networks. The ART networks learn the regularity and periodicity of the scene by observing the scene over a period of time. Thus, the ART networks allow the computer vision engine to model complex and dynamic scene backgrounds in video.
1. A computer-implemented method for generating a background model of a scene depicted in a sequence of video frames captured by a video camera, the method comprising:
receiving a video frame, wherein the video frame includes one or more appearance values for each of a plurality of pixels;
detecting a plurality of visually distinguishable patterns in the scene that appear and disappear over a period of time, the visually distinguishable patterns occurring in the one or more appearance values of each pixel over time;
evaluating using an adaptive resonance theory (ART) network the one or more appearance values of each pixel in the video frame to determine whether that pixel depicts one of a plurality of background states of the background model, wherein each background state among the plurality of background states is based on a different pattern among the plurality of visually distinguishable patterns;
classifying one or more of the pixels as depicting one of scene background based on the evaluation, and
updating the background model based on the classification.
2. The method of claim 1 , wherein evaluating using the neural network comprises:
passing the one or more appearance values for each pixel to an input layer of the ART network
mapping, by the ART network, the one or more appearance values to one of one or more clusters of the ART network; and
wherein classifying is further based on determining that a cluster to which the one or more appearance values are mapped to is one of one or more matured clusters in the ART network and wherein each matured cluster corresponds to one of the background states of the background model.
3. The method of claim 2 , wherein mapping comprises generating a new cluster in the ART network.
4. The method of claim 2 , further comprising, decaying a first cluster out of the ART network in response to determining the first cluster has not been reinforced by the mapping of the one or more appearance values to the first cluster.
5. The method of claim 2 , further comprising merging two or more clusters in the ART network.
6. The method of claim 2 , wherein mapping comprises evaluating the one or more appearance values according to a vigilance test and a choice test.
7. The method of claim 6 , wherein the one or more appearance values comprise a set of RGB color values input to the ART network and the choice test comprises determining a Euclidean distance between the RGB color values input to the ART network and a prototype set of RGB values associated with the cluster.
8. The method of claim 6 , wherein the one or more appearance values comprise a set of RGB color values input to the ART network and the vigilance test comprises determining a cosine angle and a prototype set of RGB values associated with the cluster, relative to an origin of the RGB space.
9. The method of claim 1 , wherein the one or more appearance values for each pixel comprise a set of RGB color values.
10. The method of claim 1 , wherein the visually distinguishable patterns include at least a first pattern and a second pattern that appear and disappear repeatedly.
11. The method of claim 1 , wherein the visually distinguishable patterns include a non-repeating series of different patterns that are similar in appearance.
12. The method of claim 1 , wherein the visually distinguishable patterns include a pattern that appears and persists for an extended period of time.
13. A non-transitory computer-readable storage medium containing a program, which when executed on a processor, performs an operation for generating a background model of a scene depicted in a sequence of video frames captured by a video camera, the operation comprising:
receiving a video frame, wherein the video frame includes one or more appearance values for each of a plurality of pixels;
detecting a plurality of visually distinguishable patterns in the scene that appear and disappear over a period of time, the visually distinguishable patterns occurring in the one or more appearance values of each pixel over time;
evaluating using an adaptive resonance theory (ART) network the one or more appearance values of each pixel in the video frame to determine whether that pixel depicts one of a plurality of background states of the background model, wherein each background state among the plurality of background states is based on a different pattern among the plurality of visually distinguishable patterns;
classifying one or more of the pixels as depicting one of scene background based on the evaluation; and
updating the background model based on the classification.
14. The computer-readable storage medium of claim 13 , wherein evaluating using the neural network comprises:
passing the one or more appearance values for each pixel to an input layer of the ART network,
mapping, by the ART network, the one or more appearance values to one of one or more clusters of the ART network; and
wherein classifying is further based on determining that a cluster to which the one or more appearance values are mapped to is one of one or more matured clusters in the ART network and wherein each matured cluster corresponds to one of the background states of the background model.
15. The computer-readable storage medium of claim 14 , wherein mapping comprises generating a new cluster in the ART network.
16. The computer-readable storage medium of claim 14 , wherein the operation further comprises, decaying a first cluster out of the ART network in response to determining the first cluster has not been reinforced by the mapping of the one or more appearance values to the first cluster.
17. The computer-readable storage medium of claim 14 , wherein the operation further comprises merging two or more clusters in the ART network.
18. The computer-readable storage medium of claim 14 , wherein mapping comprises evaluating the one or more appearance values according to a vigilance test and a choice test.
19. The computer-readable storage medium of claim 18 , wherein the one or more appearance values comprise a set of RGB color values input to the ART network and the choice test comprises determining a Euclidean distance between the RGB color values input to the ART network and a prototype set of RGB values associated with the cluster.
20. The computer-readable storage medium of claim 18 , wherein the one or more appearance values comprise a set of RGB color values input to the ART network and the vigilance test comprises determining a cosine angle and a prototype set of RGB values associated with the cluster, relative to an origin of the RGB space.
21. The computer-readable storage medium of claim 13 , wherein the visually distinguishable patterns include at least a first pattern and a second pattern that appear and disappear repeatedly.
22. The computer-readable storage medium of claim 13 , wherein the visually distinguishable patterns include a non-repeating series of different patterns that are similar in appearance.
23. The computer-readable storage medium of claim 13 , wherein the visually distinguishable patterns include a pattern that appears and persists for an extended period of time.
24. A system, comprising:
a video input source configured to provide a sequence of video frames, each depicting a scene;
a processor; and
a memory containing a program, which, when executed on the processor is configured to perform an operation for generating a background model of a scene depicted in a sequence of video frames captured by a video camera, the operation comprising:
receiving a video frame, wherein the video frame includes one or more appearance values for each of a plurality of pixels;
detecting a plurality of visually distinguishable patterns in the scene that appear and disappear over a period of time, the visually distinguishable patterns occurring in the one or more appearance values of each pixel over time;
evaluating using an adaptive resonance theory (ART) network the one or more appearance values of each pixel in the video frame to determine whether that pixel depicts one of a plurality of background states of the background model, wherein each background state among the plurality of background states is based on a different pattern among the plurality of visually distinguishable patterns;
classifying one or more of the pixels as depicting one of scene background based on the evaluation, and
updating the background model based on the classification.
25. The system of claim 24 , wherein evaluating using the neural network comprises:
passing the one or more appearance values for each pixel to an input layer of the ART network;
mapping, by the ART network, the one or more appearance values to one of one or more clusters of the ART network; and
wherein classifying is further based on determining that a cluster to which the one or more appearance values are mapped to is one of one or more matured clusters in the ART network and wherein each matured cluster corresponds to one of the background states of the background model.
26. The system of claim 25 , wherein mapping comprises generating a new cluster in the ART network.
27. The system of claim 25 , wherein the operation further comprises, decaying a first cluster out of the ART network in response to determining the first cluster has not been reinforced by the mapping of the one or more appearance values to the first cluster.
28. The system of claim 25 , wherein the operation further comprises merging two or more clusters in the ART network.
29. The system of claim 25 , wherein mapping comprises evaluating the one or more appearance values according to a vigilance test and a choice test.
30. The system of claim 29 , wherein the one or more appearance values comprise a set of RGB color values input to the ART network and the choice test comprises determining a Euclidean distance between the RGB color values input to the ART network and a prototype set of RGB values associated with the cluster.
31. The system of claim 29 , wherein the one or more appearance values comprise a set of RGB color values input to the ART network and the vigilance test comprises determining a cosine angle and a prototype set of RGB values associated with the cluster, relative to an origin of the RGB space.
32. The system of claim 24 , wherein the visually distinguishable patterns include at least a first pattern and a second pattern that appear and disappear repeatedly.
33. The system of claim 24 , wherein the visually distinguishable patterns include a non-repeating series of different patterns that are similar in appearance.
34. The system of claim 24 , wherein the visually distinguishable patterns include a pattern that appears and persists for an extended period of time.
35. A computer-implemented method for generating a background model of a scene depicted in a sequence of video frames captured by a video camera where visually distinguishable patterns appear and disappear over time in the scene, the method comprising:
receiving the sequence of video frames, each video frame among the sequence of video frames comprising a plurality of pixels and appearance values for each pixel among the plurality of pixels;
using a machine learning engine including an adaptive resonance theory (ART) network to detect a plurality of visually distinguishable patterns occurring in the appearance values of each pixel over time, each pattern among the plurality of visually distinguishable patterns corresponding to one cluster of appearance values observed for a corresponding pixel among the plurality of pixels over time;
incorporating clusters of appearance values into the background model to which a minimum number of appearance values for a corresponding pixel have mapped over time; and
removing clusters of appearance values from the background model that are not reinforced by subsequent appearance values for a corresponding pixel within a specified period of time.
36. A computer-implemented method for generating a background model of a scene depicted in a sequence of video frames captured by a video camera where visually distinguishable patterns appear and disappear over time in the scene, the method comprising:
receiving a video frame, wherein the video frame includes appearance values for each of a plurality of pixels;
evaluating using a neural network the appearance values of each pixel in the video frame using previously observed appearance values for that pixel obtained from preceding video frames in the sequence of video frames; and
classifying each pixel in the video frame as depicting one of scene background whose appearance values in the video frame map to mature clusters of the previously observed appearance values for that pixel in the background model, each mature cluster representing a distinct background state of a corresponding pixel in the background model, wherein each mature cluster is described by a mean and a variance, and wherein the mean and the variance of each mature cluster define metrics that determine whether the appearance values of a given pixel in the video frame map to that mature cluster.