Multi-cue object detection and analysis
Foreground objects of interest are distinguished from a background model by dividing a region of interest of a video data image into a grid array of individual cells. Each of the cells are labeled as foreground if accumulated edge energy within the cell meets an edge energy threshold, or if color intensities for different colors within each cell differ by a color intensity differential threshold, or as a function of combinations of said determinations.
1. A computer-implemented method for distinguishing foreground objects of interest from a background model, the method comprising executing on a processing unit the steps of:
acquiring frame image data for each of a plurality of individual cells in a grid array that defines a region of interest of a video data image;
for each of the cells, determining a first foreground value as true in response to determining that a color intensity of the acquired frame image data for the cell of one color is greater than a color intensity of the acquired frame image data of another color by a color intensity differential threshold;
for each of the cells, determining a second foreground value as true in response to an accumulated energy of edges detected within the acquired frame image data for the cell meeting an edge energy threshold;
labelling each of the cells as foreground in response to a true value output of a combination rule applied to the first foreground values and the second foreground values of the cells, wherein the combination rule is selected from the group consisting of a logical AND operation, a logical OR operation and a differentially weighted comparison operation; and
using the frame image data from the cells labeled as foreground to define a foreground object.
2. The method of claim 1 , further comprising:
integrating computer-readable program code into a computer system comprising the processing unit, a computer readable memory and a computer readable hardware storage device; and
wherein the computer readable program code is embodied on the computer readable hardware storage device and comprises instructions for execution by the processing unit via the computer readable memory that cause the processing unit to perform the steps of acquiring the frame image data for each of the plurality of individual cells, determining the first foreground values and the second foreground values for each of the cells, labelling each of the cells as foreground in response to the true value output of the combination rule, and using the frame image data from the cells labeled as foreground to define the foreground object.
3. The method of claim 1 , wherein the step of determining for each of the cells the second foreground value comprises:
generating a combination edge energy logical output value as an output of another combination rule applied to a plurality of accumulated energy outputs of multiple different edge detection processes for the each cell, wherein the another combination rule is selected from the group consisting of a logical AND operation and a logical OR operation; and
determining the second foreground value as the generated combination edge energy logical output value.
4. The method of claim 1 , wherein the combination rule of the step of labelling each of the cells as foreground is the differentially weighted comparison operation;
wherein the first and the second foreground values are differentially weighted; and
wherein the step of labelling each of the cells as foreground is in response to a true value of a more heavily weighted one of the first and the second foreground values.
5. The method of claim 1 , further comprising:
applying a Sobel operator to each cell to create an edge patch; and
determining for each cell the second foreground value as true in response to determining a value of “1”, and as false in response to determining a value of “0,” according to:
f
e
(
c
m
)
=
{
1
∑
i
=
1
N
E
(
x
i
)
>
t
e
0
∑
i
=
1
N
E
(
x
i
)
≤
t
e
;
wherein c m is an m th cell of the cells, E(x i ) is an energy of the edge patch at pixel x i of an “N” plurality of pixels “i” of the cell c m , and cell c m is relabeled as “1” to signify foreground in response to determining that the summation of edge energy of the N pixels of the cell c m is bigger than an edge energy threshold t e .
6. The method of claim 5 , wherein the different colors of the determined color intensities are one of red, relative green and blue color channels.
7. The method of claim 6 , further comprising:
determining for each cell the first foreground value as true in response to determining a value of “1”, and as false in response to determining a value of “0”, according to:
f
c
(
c
m
)
=
{
0
min
(
D
rg
,
D
rb
,
D
gb
)
max
(
D
rg
,
D
rb
,
D
gb
)
>
t
c
1
min
(
D
rg
,
D
rb
,
D
gb
)
max
(
D
rg
,
D
rb
,
D
gb
)
≤
t
c
}
wherein c m is an m th cell of the cells, D rg is determined as equal to “Σ i=1 N |d rb,i |”, d rb,i is an intensity difference between the red channel and the relative green channel for each pixel “i” of “N” pixels of the cell c m , D rb is determined as equal to “Σ i=1 N |d rb,i |”, d rb,i is an intensity difference between the red channel and the blue channel for each of the “N” pixels “i” of the cell c m , D gb is determined as equal to “Σ i=1 N |d rb,i |”, d rb,i is an intensity difference between the relative green channel and the blue channel for each of the “N” pixels “i” of the cell c m , wherein the value of “1” is generated in response to determining that a value of “min(D rg , D rb , D gb )” divided by “max(D rg , D rb , D gb )” is less than or equal to a color intensity differential threshold “t c ”, and the value of “0” is generated in response to determining that a value of “min(D rg , D rb , D gb )” divided by “max(D rg , D rb , D gb )” is greater than the color intensity differential threshold “t c ”.
8. The method of claim 6 , further comprising:
determining for each cell a third foreground value as true in response to determining a value of “1”, and as false in response to determining a value of “0”, according to:
f
c
1
(
c
m
)
=
{
0
min
(
V
rgb
,
{
V
rgb
′
1
…
V
rgb
′
K
}
)
max
(
V
rgb
,
{
V
rgb
′
1
…
V
rgb
′
K
}
)
>
t
c
1
1
min
(
V
rgb
,
{
V
rgb
′
1
…
V
rgb
′
K
}
)
max
(
V
rgb
,
{
V
rgb
′
1
…
V
rgb
′
K
}
)
≤
t
c
1
wherein V rgb is determined as equal to the expression “(υ 1 , υ 2 , υ 3 )′” and is a summation of intensity values for each of the individual red, relative green and blue color channels, {∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥} is related to a set of “K” adjacent cells, wherein the value of “0” is generated in response to determining that a value of “min(∥V rgb ∥,{∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥})” divided by “min(∥V rgb ∥,{∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥})” is greater than a color consistency threshold value (t c1 ), and the value of “1” is generated in response to determining that a value of “min(∥V rgb ∥,{∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥})” divided by “min(∥V rgb ∥,{∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥})” is less than or equal to the color consistency threshold value (t c1 ); and
labelling each of the cells as foreground in response to true values of at least two of the first foreground value, the second foreground value and the third foreground value.
9. A system, comprising:
a processing unit;
a computer readable memory in communication with the processing unit; and
a computer-readable hardware storage device in communication with the processing unit;
wherein the processing unit executes program instructions stored on the computer-readable hardware storage device via the computer readable memory and thereby:
acquires frame image data for each of a plurality of individual cells in a grid array that defines a region of interest of a video data image;
for each of the cells, determines a first foreground value as true in response to determining that a color intensity of the acquired frame image data for the cell of one color is greater than a color intensity of the acquired frame image data of another color by a color intensity differential threshold;
for each of the cells, determines a second foreground value as true in response to an accumulated energy of edges detected within the acquired frame image data for the cell meeting an edge energy threshold;
labels each of the cells as foreground in response to a true value output of a combination rule applied to the first foreground values and the second foreground values of the cells, wherein the combination rule is selected from the group consisting of a logical AND operation, a logical OR operation and a differentially weighted comparison operation; and
uses the frame image data from the cells labeled as foreground to define a foreground object.
10. The system of claim 9 , wherein the processing unit executes the program instructions stored on the computer-readable hardware storage device via the computer readable memory and thereby:
generates a combination edge energy logical output value as an output of another combination rule applied to a plurality of accumulated energy outputs of multiple different edge detection processes for each cell, wherein the another combination rule is selected from the group consisting of a logical AND operation and a logical OR operation; and
determines the second foreground value as the generated combination edge energy logical output value.
11. The system of claim 9 , wherein the combination rule of the step of labelling each of the cells as foreground is the differentially weighted comparison operation;
wherein the first and the second foreground values are differentially weighted; and
wherein the processing unit executes the program instructions stored on the computer-readable hardware storage device via the computer readable memory and thereby labels each of the cells as foreground is in response to a true value of a more heavily weighted one of the first and the second foreground values.
12. The system of claim 9 , wherein the processing unit executes the program instructions stored on the computer-readable hardware storage device via the computer readable memory and thereby:
applies a Sobel operator to each of the cells to create an edge patch; and
determines for each cell the second foreground value as true in response to determining a value of “1”, and as false in response to determining a value of “0,” according to:
f
e
(
c
m
)
=
{
1
∑
i
=
1
N
E
(
x
i
)
>
t
e
0
∑
i
=
1
N
E
(
x
i
)
≤
t
e
;
wherein c m is an m th cell of the cells, E(x i ) is an energy of the edge patch at pixel x i of an “N” plurality of pixels “i” of the cell c m , and cell c m is relabeled as “1” to signify foreground in response to determining that the summation of edge energy of the N pixels of the cell c m is bigger than an edge energy threshold t e .
13. The system of claim 12 , wherein the processing unit executes the program instructions stored on the computer-readable hardware storage device via the computer readable memory and thereby:
determines for each cell the first foreground value as true in response to determining a value of “1”, and as false in response to determining a value of “0”, according to:
f
c
(
c
m
)
=
{
0
min
(
D
rg
,
D
rb
,
D
gb
)
max
(
D
rg
,
D
rb
,
D
gb
)
>
t
c
1
min
(
D
rg
,
D
rb
,
D
gb
)
max
(
D
rg
,
D
rb
,
D
gb
)
≤
t
c
}
wherein c m is an m th cell of the cells, D rg is determined as equal to “Σ i=1 N |d rb,i |”, d rb,i is an intensity difference between the red channel and the relative green channel for each pixel “i” of “N” pixels of the cell c m , D rb is determined as equal to “Σ i=1 N |d rb,i |”, d rb,i is an intensity difference between the red channel and the blue channel for each of the “N” pixels “i” of the cell c m , D gb is determined as equal to “Σ i=1 N |d rb,i |”, d rb,i is an intensity difference between the relative green channel and the blue channel for each of the “N” pixels “i” of the cell c m , wherein the value of “1” is generated in response to determining that a value of “min(D rg , D rb , D gb )” divided by “max(D rg , D rb , D gb )” is less than or equal to a color intensity differential threshold “t c ”, and the value of “0” is generated in response to determining that a value of “min(D rg , D rb , D gb )” divided by “max(D rg , D rb , D gb )” is greater than the color intensity differential threshold “t c ”.
14. The system of claim 13 , wherein the processing unit executes the program instructions stored on the computer-readable hardware storage device via the computer readable memory and thereby:
determines for each cell a third foreground value as true in response to determining a value of “1”, and as false in response to determining a value of “0”, according to:
f
c
1
(
c
m
)
=
{
0
min
(
V
rgb
,
{
V
rgb
′
1
…
V
rgb
′
K
}
)
max
(
V
rgb
,
{
V
rgb
′
1
…
V
rgb
′
K
}
)
>
t
c
1
1
min
(
V
rgb
,
{
V
rgb
′
1
…
V
rgb
′
K
}
)
max
(
V
rgb
,
{
V
rgb
′
1
…
V
rgb
′
K
}
)
≤
t
c
1
wherein V rgb is determined as equal to the expression “(υ 1 , υ 2 , υ 3 )′” and is a summation of intensity values for each of the individual red, relative green and blue color channels, {∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥} is related to a set of “K” adjacent cells, wherein the value of “0” is generated in response to determining that a value of “min(∥V rgb ∥,{∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥})” divided by “min(∥V rgb ∥,{∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥})” is greater than a color consistency threshold value (t c1 ), and the value of “1” is generated in response to determining that a value of “min(∥V rgb ∥,{∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥})” divided by “min(∥V rgb ∥,{∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥})” is less than or equal to the color consistency threshold value (t c1 ); and
labels each of the cells as foreground in response to true values of at least two of the first foreground value, the second foreground value and the third foreground value.
15. An article of manufacture, comprising:
a computer readable hardware storage device having computer readable program code embodied therewith, wherein the computer readable hardware storage device is not transitory per se, and computer readable program code comprises instructions for execution by a computer processing unit that cause the computer processing unit to:
acquire frame image data for each of a plurality of individual cells in a grid array that defines a region of interest of a video data image;
for each of the cells, determine a first foreground value as true in response to determining that a color intensity of the acquired frame image data for the cell of one color is greater than a color intensity of the acquired frame image data of another color by a color intensity differential threshold;
for each of the cells, determine a second foreground value as true in response to an accumulated energy of edges detected within the acquired frame image data for the cell meeting an edge energy threshold;
label each of the cells as foreground in response to a true value output of a combination rule applied to the first foreground values and the second foreground values of the cells, wherein the combination rule is selected from the group consisting of a logical AND operation, a logical OR operation and a differentially weighted comparison operation; and
use the frame image data from the cells labeled as foreground to define a foreground object.
16. The article of manufacture of claim 15 , wherein the computer readable program code instructions cause the computer processing unit to:
generate a combination edge energy logical output value as an output of another combination rule applied to a plurality of accumulated energy outputs of multiple different edge detection processes for each cell, wherein the another combination rule is selected from the group consisting of a logical AND operation and a logical OR operation; and
determine the second foreground value as the generated combination edge energy logical output value.
17. The article of manufacture of claim 15 , wherein the combination rule of the step of labelling each of the cells as foreground is the differentially weighted comparison operation;
wherein the first and the second foreground values are differentially weighted; and
wherein the computer readable program code instructions cause the computer processing unit to label each of the cells as foreground is in response to a true value of a more heavily weighted one of the first and the second foreground values.
18. The article of manufacture of claim 15 , wherein the computer readable program code instructions cause the computer processing unit to:
apply a Sobel operator to each of the cells to create an edge patch; and
determine for each cell the second foreground value as true in response to determining a value of “1”, and as false in response to determining a value of “0,” according to:
f
e
(
c
m
)
=
{
1
∑
i
=
1
N
E
(
x
i
)
>
t
e
0
∑
i
=
1
N
E
(
x
i
)
≤
t
e
;
wherein c m is an m th cell of the cells, E(x i ) is an energy of the edge patch at pixel x i of an “N” plurality of pixels “i” of the cell c m , and cell c m is relabeled as “1” to signify foreground in response to determining that the summation of edge energy of the N pixels of the cell c m is bigger than an edge energy threshold t e .
19. The article of manufacture of claim 18 , wherein the computer readable program code instructions cause the computer processing unit to:
determine for each cell the first foreground value as true in response to determining a value of “1”, and as false in response to determining a value of “0”, according to:
f
c
(
c
m
)
=
{
0
min
(
D
rg
,
D
rb
,
D
gb
)
max
(
D
rg
,
D
rb
,
D
gb
)
>
t
c
1
min
(
D
rg
,
D
rb
,
D
gb
)
max
(
D
rg
,
D
rb
,
D
gb
)
≤
t
c
}
wherein c m is an m th cell of the cells, D rg is determined as equal to “Σ i=1 N |d rb,i |”, d rb,i is an intensity difference between the red channel and the relative green channel for each pixel “i” of “N” pixels of the cell c m , D rb is determined as equal to “Σ i=1 N |d rb,i |”, d rb,i is an intensity difference between the red channel and the blue channel for each of the “N” pixels “i” of the cell c m , D gb is determined as equal to “Σ i=1 N |d rb,i |”, d rb,i is an intensity difference between the relative green channel and the blue channel for each of the “N” pixels “i” of the cell c m , wherein the value of “1” is generated in response to determining that a value of “min(D rg , D rb , D gb )” divided by “max(D rg , D rb , D gb )” is less than or equal to a color intensity differential threshold “t c ”, and the value of “0” is generated in response to determining that a value of “min(D rg , D rb , D gb )” divided by “max(D rg , D rb , D gb )” is greater than the color intensity differential threshold “t c ”.
20. The article of manufacture of claim 19 , wherein the computer readable program code instructions cause the computer processing unit to:
determine for each cell a third foreground value as true in response to determining a value of “1”, and as false in response to determining a value of “0”, according to:
f
c
1
(
c
m
)
=
{
0
min
(
V
rgb
,
{
V
rgb
′
1
…
V
rgb
′
K
}
)
max
(
V
rgb
,
{
V
rgb
′
1
…
V
rgb
′
K
}
)
>
t
c
1
1
min
(
V
rgb
,
{
V
rgb
′
1
…
V
rgb
′
K
}
)
max
(
V
rgb
,
{
V
rgb
′
1
…
V
rgb
′
K
}
)
≤
t
c
1
wherein V rgb is determined as equal to the expression “(υ 1 , υ 2 , υ 3 )′” and is a summation of intensity values for each of the individual red, relative green and blue color channels, {∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥} is related to a set of “K” adjacent cells, wherein the value of “0” is generated in response to determining that a value of “min(∥V rgb ∥,{∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥})” divided by “min(∥V rgb ∥,{∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥})” is greater than a color consistency threshold value (t c1 ), and the value of “1” is generated in response to determining that a value of “min(∥V rgb ∥,{∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥})” divided by “min(∥V rgb ∥,{∥V′ rgb 1 ∥ . . . ∥V′ rgb K ∥})” is less than or equal to the color consistency threshold value (t c1 ); and
label each of the cells as foreground in response to true values of at least two of the first foreground value, the second foreground value and the third foreground value.