Adaptive update of background pixel thresholds using sudden illumination change detection
View Patent ↗Techniques are disclosed for a computer vision engine to update both a background model and thresholds used to classify pixels as depicting scene foreground or background in response to detecting that a sudden illumination changes has occurred in a sequence of video frames. The threshold values may be used to specify how much pixel a given pixel may differ from corresponding values in the background model before being classified as depicting foreground. When a sudden illumination change is detected, the values for pixels affected by sudden illumination change may be used to update the value in the background image to reflect the value for that pixel following the sudden illumination change as well as update the threshold for classifying that pixel as depicting foreground/background in subsequent frames of video.
1. A computer-implemented method for analyzing a sequence of video frames depicting a scene captured by a video camera, the method comprising:
classifying each of a plurality of pixels in a first frame of the sequence of video frames as depicting one of scene background and scene foreground, wherein each pixel is classified based on a comparison of (i) a relative deviation determined using a pixel value from the first frame and a corresponding pixel value in a background image and (ii) a classification threshold assigned to each pixel in the background image; and
upon determining that a group of pixels in the first frame were classified as depicting scene foreground as a result of a sudden illumination change (SIC):
for each pixel in the group of pixels:
re-classifying the pixel as depicting scene background, and
decreasing the classification threshold assigned to the pixel to a value below the relative deviation determined for the pixel; and
for one or more successive frames of video following the first frame:
updating the pixel values in the background image corresponding to the group of pixels, based on pixel values obtained from video frames following the first frame, and
increasing the classification threshold assigned to pixels in the background image for pixels corresponding to the group of pixels.
2. The computer-implemented method of claim 1 , wherein the pixel values in the background image represent a pixel value expected to be present in a frame of video captured by the video camera when the pixel depicts scene background.
3. The computer-implemented method of claim 1 , further comprising:
prior to classifying each of the plurality of pixels in the first frame of the sequence of video frames as depicting one of scene background and scene foreground, assigning an initial threshold to the pixel values in the background image based on a camera noise model and an initial radiance of a respective pixel in the background image.
4. The computer-implemented method of claim 1 , wherein classifying each of the plurality of pixels in the first frame of the sequence of video frames as depicting one of scene background and scene foreground, comprises:
determining the relative deviation as ρ R , ρ G , and ρ B between a first pixel in the first frame and the corresponding pixel in the background image according to:
ρ
R
=
255
R
-
R
_
+
1
ρ
G
=
255
G
-
G
_
+
1
ρ
B
=
255
B
-
B
_
+
1
wherein R, G, and B represent a color channel value for the first pixel in the first frame and wherein R , G , and B each represent a corresponding pixel color channel value from the background image;
determining an overall relative difference (Δ) from the ρ R , ρ G , and ρ B values;
classifying the first pixel as depicting scene background when the overall relative difference (Δ) is above or equal to the threshold assigned to the first pixel; and
classifying the first pixel as depicting scene foreground when the overall relative difference (Δ) is below the threshold assigned to the first pixel.
5. The computer-implemented method of claim 4 , wherein the overall relative difference (Δ) for the first pixel is calculated from the ρ R , ρ G , and ρ B values according to:
Δ
=
1
2
{
ρ
R
+
ρ
G
+
ρ
B
-
minimum
(
ρ
R
,
ρ
G
,
ρ
B
)
}
.
6. The computer-implemented method of claim 1 , further comprising:
classifying one or more of the pixels in the group as being an SIC candidate pixel by evaluating a spatial neighborhood of each pixel in the group, wherein the spatial neighborhood comprises a 3×3 grid of pixels with the pixel being evaluated at the center of the 3×3 grid.
7. The computer-implemented method of claim 6 , wherein determining that the group of pixels in the first frame were classified as depicting scene foreground as the result of the sudden illumination change (SIC) comprises, determining that a majority of pixels in the group are classified as SIC candidate pixels, based on the evaluations of the spatial neighborhoods of the pixels in the group.
8. The computer-implemented method of claim 1 , wherein the pixel values in the background image corresponding to the group of pixels are updated according to the following:
bg r =α( bg r )+(1−α)(raw r ) bg g =α( bg g )+(1−α)(raw g ) bg b =α( bg b )+(1−α)(raw b )
wherein bg r , bg g , and bg b represent color channel values for a respective pixel in the background image, raw r , raw g , and raw b represent the R, G, and B color channels values for the corresponding color channel values in a current frame, and wherein αrepresents a value selected from a range of [0,1], inclusive.
9. A computer-readable storage medium containing a program which, when executed by a processor, performs an operation for analyzing a sequence of video frames depicting a scene captured by a video camera, the operation comprising:
classifying each of a plurality of pixels in a first frame of the sequence of video frames as depicting one of scene background and scene foreground, wherein each pixel is classified based on a comparison of (i) a relative deviation determined using a pixel value from the first frame and a corresponding pixel value in a background image and (ii) a classification threshold assigned to each pixel in the background image; and
upon determining that a group of pixels in the first frame were classified as depicting scene foreground as a result of a sudden illumination change (SIC):
for each pixel in the group of pixels:
re-classifying the pixel as depicting scene background, and
decreasing the classification threshold assigned to a value below the relative deviation determined for the pixel using the pixel value from the first frame and the corresponding pixel value from the background image; and
for one or more successive frames of video following the first frame:
updating the pixel values in the background image corresponding to the group of pixels, based on pixel values obtained from video frames following the first frame, and
increasing the classification threshold assigned to pixels in the background image for pixels corresponding to the group of pixels.
10. The computer-readable storage medium of claim 9 , wherein the pixel values in the background image represent a pixel value expected to be present in a frame of video captured by the video camera when the pixel depicts scene background.
11. The computer-readable storage medium of claim 9 , wherein the operation further comprises:
prior to classifying each of the plurality of pixels in the first frame of the sequence of video frames as depicting one of scene background and scene foreground, assigning an initial threshold to the pixel values in the background image based on a camera noise model and an initial radiance of a respective pixel in the background image.
12. The computer-readable storage medium of claim 9 , wherein classifying each of the plurality of pixels in the first frame of the sequence of video frames as depicting one of scene background and scene foreground, comprises:
determining the relative deviation as ρ R , ρ G , and ρ B between a first pixel in the first frame and the corresponding pixel in the background image according to:
ρ
R
=
255
R
-
R
_
+
1
ρ
G
=
255
G
-
G
_
+
1
ρ
B
=
255
B
-
B
_
+
1
wherein R, G, and B represent a color channel value for the first pixel in the first frame and wherein R , G , and B each represent a corresponding pixel color channel value from the background image;
determining an overall relative difference (Δ) from the ρ R , ρ G , and ρ B values;
classifying the first pixel as depicting scene background when the overall relative difference (Δ) is above or equal to the threshold assigned to the first pixel; and
classifying the first pixel as depicting scene foreground when the overall relative difference (Δ) is below the threshold assigned to the first pixel.
13. The computer-readable storage medium of claim 12 , wherein the overall relative difference (Δ) for the first pixel is calculated from the ρ R , ρ G , and ρ B values according to:
Δ
=
1
2
{
ρ
R
+
ρ
G
+
ρ
B
-
minimum
(
ρ
R
,
ρ
G
,
ρ
B
)
}
.
14. The computer-readable storage medium of claim 9 , wherein the operation further comprises:
classifying one or more of the pixels in the group as being an SIC candidate pixel by evaluating a spatial neighborhood of each pixel in the group, wherein the spatial neighborhood comprises a 3×3 grid of pixels with the pixel being evaluated at the center of the 3×3 grid.
15. The computer-readable storage medium of claim 14 , wherein determining that the group of pixels in the first frame were classified as depicting scene foreground as the result of the sudden illumination change (SIC) comprises, determining that a majority of pixels in the group are classified as SIC candidate pixels, based on the evaluations of the spatial neighborhoods of the pixels in the group.
16. The computer-readable storage medium of claim 9 , wherein the pixel values in the background image corresponding to the group of pixels are updated according to the following:
bg r =α( bg r )+(1−α)(raw r ) bg g =α( bg g )+(1−α)(raw g ) bg b =α( bg b )+(1−α)(raw b )
wherein bg r , bg g , and bg b represent color channel values for a respective pixel in the background image, raw r , raw g , and raw b represent the R, G, and B color channels values for the corresponding color channel values in a current frame, and wherein αrepresents a value selected from a range of [0,1], inclusive.
17. A system, comprising:
a video input source configured to provide a sequence of video frames, each depicting a scene;
a processor; and
a memory containing a program, which, when executed on the processor is configured to perform an operation for analyzing the scene, as depicted by the sequence of video frames captured by the video input source, the operation comprising:
classifying each of a plurality of pixels in a first frame of the sequence of video frames as depicting one of scene background and scene foreground, wherein each pixel is classified based on a comparison of (i) a relative deviation determined using a pixel value from the first frame and a corresponding pixel value in a background image and (ii) a classification threshold assigned to each pixel in the background image,
upon determining that a group of pixels in the first frame were classified as depicting scene foreground as a result of a sudden illumination change (SIC):
for each pixel in the group of pixels:
re-classifying the pixel as depicting scene background; and
decreasing the classification threshold assigned to a value below the relative deviation determined for the pixel using the pixel value from the first frame and the corresponding pixel value from the background image, and
for one or more successive frames of video following the first frame:
updating the pixel values in the background image corresponding to the group of pixels, based on pixel values obtained from video frames following the first frame; and
increasing the classification threshold assigned to pixels in the background image for pixels corresponding to the group of pixels.
18. The system of claim 17 , wherein the pixel values in the background image represent a pixel value expected to be present in a frame of video captured by the video camera when the pixel depicts scene background.
19. The system of claim 17 , wherein the operation further comprises:
prior to classifying each of the plurality of pixels in the first frame of the sequence of video frames as depicting one of scene background and scene foreground, assigning an initial threshold to the pixel values in the background image based on a camera noise model and an initial radiance of a respective pixel in the background image.
20. The system of claim 17 , wherein classifying each of the plurality of pixels in the first frame of the sequence of video frames as depicting one of scene background and scene foreground, comprises:
determining the relative deviation as ρ R , ρ G , and ρ B between a first pixel in the first frame and the corresponding pixel in the background image according to:
ρ
R
=
255
R
-
R
_
+
1
ρ
G
=
255
G
-
G
_
+
1
ρ
B
=
255
B
-
B
_
+
1
wherein R, G, and B represent a color channel value for the first pixel in the first frame and wherein R , G , and B each represent a corresponding pixel color channel value from the background image;
determining an overall relative difference (Δ) from the ρ R , ρ G , and ρ B values;
classifying the first pixel as depicting scene background when the overall relative difference (Δ) is above or equal to the threshold assigned to the first pixel; and
classifying the first pixel as depicting scene foreground when the overall relative difference (Δ) is below the threshold assigned to the first pixel.
21. The system of claim 20 , wherein the overall relative difference (Δ) for the first pixel is calculated from the ρ R , ρ G , and ρ B values according to:
Δ
=
1
2
{
ρ
R
+
ρ
G
+
ρ
B
-
minimum
(
ρ
R
,
ρ
G
,
ρ
B
)
}
.
22. The system of claim 17 , wherein the operation further comprises:
classifying one or more of the pixels in the group as being an SIC candidate pixel by evaluating a spatial neighborhood of each pixel in the group, wherein the spatial neighborhood comprises a 3×3 grid of pixels with the pixel being evaluated at the center of the 3×3 grid.
23. The system of claim 22 , wherein determining that the group of pixels in the first frame were classified as depicting scene foreground as the result of the sudden illumination change (SIC) comprises, determining that a majority of pixels in the group are classified as SIC candidate pixels, based on the evaluations of the spatial neighborhoods of the pixels in the group.
24. The system of claim 17 , wherein the pixel values in the background image corresponding to the group of pixels are updated according to the following:
bg r =α( bg r )+(1−α)(raw r ) bg g =α( bg g )+(1−α)(raw g ) bg b =α( bg b )+(1−α)(raw b )
wherein bg r , bg g , and bg b represent color channel values for a respective pixel in the background image, raw r , raw g , and raw b represent the R, G, and B color channels values for the corresponding color channel values in a current frame, and wherein αrepresents a value selected from a range of [0,1], inclusive.