Video surveillance system based on Gaussian mixture modeling with two-type learning rate control scheme
In the present invention, we identify that such a tradeoff between robustness to background changes and sensitivity to foreground abnormalities can be easily controlled by a new computational scheme of two-type learning rate control for the Gaussian mixture modeling (GMM). Based on the proposed rate control scheme, a new video surveillance system that applies feedbacks of pixel properties computed in object-level analysis to the learning rate controls of the GMM in pixel-level background modeling is developed. Such a system gives better regularization of background adaptation and is efficient in resolving the tradeoff for many surveillance applications.
1. A video surveillance system based on Gaussian mixture modeling with a two-type learning rate control scheme comprising:
a processor configured to execute at least three processing modules, stored in an non-transitory storage medium, of:
a background model maintenance module that constructs a background model as its module output from an image frame input, based on the Gaussian mixture modeling with a two-type learning rate control scheme;
a foreground pixel identification module that marks a plurality of pixels of a current image frame of the plurality of image frames as foreground or background, and generates a foreground map as its output; and
a high-level information extraction module that classifies the foreground pixels into a plurality of object level types via foreground object analysis; and
at least two links of:
an input link of the plurality of image frames; and
a feedback link of high-level information that is generated by the high-level information extraction module and transmitted to the background model maintenance module,
wherein the background model maintenance module being operated to perform steps of:
receiving one of the plurality of image frames It;
matching each pixel intensity It,x to one of N Gaussian models of a corresponding Gaussian mixture model, based on a weight-based model matching rule,
l
(
t
,
x
)
=
arg
min
n
=
1
,
…
,
N
d
t
,
x
,
n
,
subject
to
d
t
,
x
,
n
=
{
-
w
t
-
1
,
x
,
n
if
I
t
,
x
-
μ
t
-
1
,
x
,
n
≤
T
σ
σ
t
-
1
,
x
,
n
inf
otherwise
,
where l(t,x) indexes the best matched Gaussian model, if existing, for I 1,x , the three parameters, μ t-1,x,n , σ t-1,x,n , and w t-1,x,n , denote the mean, standard deviation and mixture weight, respectively, of the nth Gaussian distribution of the Gaussian mixture model, and T g is a given threshold;
applying the two-type learning rate control scheme to the updating of the Gaussian mixture model, wherein two independent learning rates, ρ t,x and η t,x , are adopted by the iterative updating rules of the Gaussian parameters (μ and σ) and of the mixture weight (w), respectively, for the Gaussian mixture model of the location x at the current time instance t; and
producing the Gaussian mixture background model at the current time instance t as the module output.
2. The video surveillance system of claim 1 , wherein the foreground pixel identification module being operated to perform steps of:
receiving the Gaussian mixture background model at the current time instance t;
marking each image pixel as foreground or background by the mixture weight thresholding,
F
t
,
x
=
{
0
if
w
t
,
x
,
l
≥
T
w
,
(
Background
)
1
otherwise
,
(
Foreground
)
wherein F t,x is a binary, foreground/background indicator for a pixel located at x at the time instance t, is the mixture weight of the l(t,x)th Gaussian model matching the image intensity I t,x and T w is a given weight threshold; and
producing a foreground map F t ={F t,x |∀x} at the current time instance t as its module output, wherein the foreground map F t is a set of foreground/background indicators F t,x s for all pixel locations xs.
3. The video surveillance system of claim 2 , wherein the high-level information extraction module being operated to perform steps of:
receiving the foreground map F t at the current time instance t;
deriving a general-purposed pixel type indicator O t,x for each of the image pixels via manual specification and/or automatic pixel type analysis;
producing a pixel type map O t ={O t,x |∀x} at the current time instance as its module output, wherein the pixel, type map O t is a set of pixel type indicators O t,x s for all pixel locations xs; and
feeding the derived high-level information comprising the pixel type map back to the background model maintenance module.
4. The video surveillance system of claim 1 , wherein the learning rate η t,x for regularizing the updating speed of mixture weight, for a pixel located at x at the time instance t, in the two-type learning rate control scheme is designed as an independent rate control from the other learning rate ρ t , for regularizing the updating speed of Gaussian parameters.
5. The video surveillance system of claim 1 , wherein the learning rate η t,x for regularizing the updating speed of mixture weight, for a pixel located at x at the time instance t, in the two-type learning rate control scheme can be
correlated to the high-level information of pixel type computed at previous time instances; and
defined as a function η(β) that adopts a parameter β in a form of scalar, vector or matrix as its function argument for controlling its value.
6. The video surveillance system of claim 3 , wherein general-purposed pixel type indicator O t,x generated by the high-level information extraction module for the learning rate control of the Gaussian mixture modeling can be further specified as:
two types of foreground and background for the basic learning rate control for the Gaussian mixture modeling;
four commonly-seen types of background, shadow, still foreground, and moving foreground that can be specifically denoted by
O
t
,
x
=
{
0
if
Type
(
I
t
,
x
)
=
Background
1
if
Type
(
I
t
,
x
)
=
Shadow
,
2
if
Type
(
I
t
,
x
)
=
Still
foreground
,
3
if
Type
(
I
t
,
x
)
=
Moving
foreground
for four-class learning rate control for the Gaussian mixture modeling; and
user-assigned types for some image locations in real-time system processing or prior to system processing.
7. The video surveillance system of claim 5 , wherein the learning rate control of η t,x being correlated to the high-level information of the pixel type of location x at the time instance t can be explicitly formulated as plural number of rate control sub-equations corresponding to different assignments of the pixel type indicator O t,x for flexible learning rate control.
8. The video surveillance system of claim 7 , wherein the learning rate control of η being correlated to the four commonly-seen types of background, shadow, still foreground, and moving foreground can be further formulated as
η
t
,
x
(
β
)
=
{
(
1
-
β
b
)
η
t
-
1
,
x
+
η
b
β
b
if
O
t
-
1
,
x
=
0
,
β
d
N
(
I
t
,
x
;
μ
t
-
1
,
x
,
b
(
t
,
x
)
,
σ
t
-
1
,
x
,
b
(
t
,
x
)
2
)
if
O
t
-
1
,
x
=
1
,
β
s
if
O
t
-
1
,
x
=
2
,
β
m
if
O
t
-
1
,
x
=
3
where β=[βbβdβsβm] T ε 4 is the parameter in vector form of the learning rate function 77 for tuning the rate settings of the corresponding pixel types, η b is a preset constant for the rate formulation of the background type, b(t,x) is the index of the most probable background model defined as
b
(
t
,
x
)
=
arg
max
n
=
1
,
…
,
N
w
t
,
x
,
n
,
and N(●; μ, σ 2 ) is a Gaussian distribution with model mean μ and standard deviation σ.