Hierarchical sudden illumination change detection using radiance consistency within a spatial neighborhood
View Patent ↗Techniques are disclosed for detecting sudden illumination changes using radiance consistency within a spatial neighborhood. A background/foreground (BG/FG) component of a behavior recognition system may be configured to generate a background image depicting a scene background. Further, the (BG/FG) component may periodically evaluate a current video frame to determine whether a sudden illumination change has occurred. A sudden illumination change occurs when scene lighting changes dramatically from one frame to the next (or over a small number of frames).
1. A computer-implemented method for detecting a sudden illumination change (SIC) in a sequence of video frames, comprising:
receiving a current video frame, of the sequence of video frames;
identifying one or more sets of pixels in the current video frame, wherein each set of pixels is classified as depicting a foreground object;
for each pixel in at least a first one of the sets of pixels:
identifying a radiance value for the pixel and for each pixel within a spatial neighborhood of the pixel, and
determining, for the pixel, whether the radiance values for the pixel and the pixels in the spatial neighborhood indicate that a sudden illumination change has occurred in a scene depicted by the sequence of video frames; and
upon determining that a majority of pixels in the first set of pixels indicate that a sudden illumination change has occurred, reclassifying one or more pixels in the first set of pixels as depicting a background of the scene in the current frame, wherein the reclassified pixels were classified as depicting scene background in one or more frames prior to the current frame.
2. The computer-implemented method of claim 1 , further comprising:
updating a set of pixels in a background image of the scene corresponding to the pixels in the first set of pixels.
3. The computer-implemented method of claim 1 , wherein the spatial neighborhood for a given pixel comprises a 3×3 grid of pixels radiance values with a radiance value given pixel in the center of the 3×3 grid.
4. The computer-implemented method of claim 1 , wherein determining, for the pixel, whether the radiance values for the pixel and the pixels in the spatial neighborhood indicate that a sudden illumination change has occurred, comprises:
identifying, in a background image, a pixel corresponding to the pixel in the current video frame;
determining an intensity difference between the corresponding pixel and each pixel in a spatial neighborhood of the corresponding pixel;
determining an intensity difference between the pixel and each pixel in the spatial neighborhood of the pixel in the current video frame;
normalizing the intensity differences determined between the pixel and each pixel in the spatial neighborhood of the pixel in the current video frame;
determining an absolute difference between each respective pixel in the spatial neighborhood of the pixel in the current video frame and each respective pixel in the spatial neighborhood of the corresponding pixel;
for each pixel in the spatial neighborhood of the pixel in the current video frame, determining whether the absolute difference is below a first specified threshold, and
if so, incrementing a counter representing how many pixels in the spatial neighborhood are below the first specified threshold;
upon determining the counter exceeds a second threshold, setting a status of the pixel as being an SIC candidate pixel, indicating that the pixel was classified as depicting a scene foreground object due to a sudden illumination change in the scene depicted by the sequence of video frames.
5. The computer-implemented method of claim 4 , further comprising:
upon determining a count of pixels classified as depicting the foreground object which have a status of SIC candidate pixel is below a third threshold, setting a status of the foreground object as being a true foreground object, the status indicating that a sudden illumination change has not occurred for the pixels classified as depicting the foreground in the scene depicted by the sequence of video frames; and
upon determining a count of pixels classified as depicting the foreground object which have a status of SIC candidate pixel is above a fourth threshold, setting a status of the foreground object blob as being an SIC foreground object, the status indicating that a sudden illumination change has occurred for that foreground object in the scene in the current video frame.
6. The computer-implemented method of claim 4 , further comprising:
generating a lower-resolution image of the current video frame; and
repeating the computer-implemented method for detecting a sudden illumination change (SIC) in a sequence of video frames recited by claim 1 , using the lower resolution image as the current frame of video.
7. The computer-implemented method of claim 4 , wherein normalizing the intensity differences comprises, dividing each of the intensity differences determined between the pixel and each pixel in the spatial neighborhood of the pixel in the current video frame using a ratio of the radiance value of the pixel in the current video frame and a radiance value of the corresponding pixel in the background image.
8. The computer-implemented method of claim 4 , wherein the first specified threshold is determined using a Gaussian distribution determined using the pixel radiance value as a mean and an expected camera noise at the pixel radiance value as a variance.
9. The computer-implemented method of claim 8 , wherein the first specified threshold is three standard deviations from the mean.
10. The computer-implemented method of claim 1 , wherein each radiance value provides a grayscale value between 0 and 255.
11. A non-transitory computer-readable storage medium containing a program which, when executed by a processor, performs an operation for detecting a sudden illumination change (SIC) in a sequence of video frames, the operation comprising:
receiving a current video frame, of the sequence of video frames;
identifying one or more sets of pixels in the current video frame, wherein each set of pixels is classified as depicting a foreground object;
for each pixel in at least a first one of the sets of pixels:
identifying a radiance value for the pixel and for each pixel within a spatial neighborhood of the pixel, and
determining, for the pixel, whether the radiance values for the pixel and the pixels in the spatial neighborhood indicate that a sudden illumination change has occurred in a scene depicted by the sequence of video frames; and
upon determining that a majority of pixels in the first set of pixels indicate that a sudden illumination change has occurred, reclassifying one or more pixels in the first set of pixels as depicting a background of the scene in the current frame, wherein the reclassified pixels were classified as depicting scene background in one or more frames prior to the current frame.
12. The non-transitory computer-readable storage medium of claim 11 , wherein the operation further comprises:
updating a set of pixels in a background image of the scene corresponding to the pixels in the first set of pixels.
13. The non-transitory computer-readable storage medium of claim 11 , wherein the spatial neighborhood for a given pixel comprises a 3×3 grid of pixels radiance values with a radiance value given pixel in the center of the 3×3 grid.
14. The non-transitory computer-readable storage medium of claim 11 , wherein determining, for the pixel, whether the radiance values for the pixel and the pixels in the spatial neighborhood indicate that a sudden illumination change has occurred, comprises:
identifying, in a background image, a pixel corresponding to the pixel in the current video frame;
determining an intensity difference between the corresponding pixel and each pixel in a spatial neighborhood of the corresponding pixel;
determining an intensity difference between the pixel and each pixel in the spatial neighborhood of the pixel in the current video frame;
normalizing the intensity differences determined between the pixel and each pixel in the spatial neighborhood of the pixel in the current video frame;
determining an absolute difference between each respective pixel in the spatial neighborhood of the pixel in the current video frame and each respective pixel in the spatial neighborhood of the corresponding pixel;
for each pixel in the spatial neighborhood of the pixel in the current video frame, determining whether the absolute difference is below a first specified threshold, and
if so, incrementing a counter representing how many pixels in the spatial neighborhood are below the first specified threshold;
upon determining the counter exceeds a second threshold, setting a status of the pixel as being an SIC candidate pixel, indicating that the pixel was classified as depicting a scene foreground object due to a sudden illumination change in the scene depicted by the sequence of video frames.
15. The non-transitory computer-readable storage medium of claim 14 , wherein the operation further comprises:
upon determining a count of pixels classified as depicting the foreground object which have a status of SIC candidate pixel is below a third threshold, setting a status of the foreground object as being a true foreground object, the status indicating that a sudden illumination change has not occurred for the pixels classified as depicting the foreground in the scene depicted by the sequence of video frames; and
upon determining a count of pixels classified as depicting the foreground object which have a status of SIC candidate pixel is above a fourth threshold, setting a status of the foreground object blob as being an SIC foreground object, the status indicating that a sudden illumination change has occurred for that foreground object in the scene in the current video frame.
16. The non-transitory computer-readable storage medium of claim 14 , wherein normalizing the intensity differences comprises, dividing each of the intensity differences determined between the pixel and each pixel in the spatial neighborhood of the pixel in the current video frame using a ratio of the radiance value of the pixel in the current video frame and a radiance value of the corresponding pixel in the background image.
17. The non-transitory computer-readable storage medium of claim 14 , wherein the first specified threshold is determined using a Gaussian distribution determined using the pixel radiance value as a mean and an expected camera noise at the pixel radiance value as a variance.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the first specified threshold is three standard deviations from the mean.
19. A system comprising,
a video input source configured to provide a sequence of video frames, each depicting a scene;
a processor; and a memory containing a program, which, when executed on the processor is configured to perform an operation for detecting a sudden illumination change (SIC) in a sequence of video frames by performing the steps of:
receiving a current video frame, of the sequence of video frames,
identifying one or more sets of pixels in the current video frame, wherein each set of pixels is classified as depicting a foreground object,
for each pixel in at least a first one of the sets of pixels:
identifying a radiance value for the pixel and for each pixel within a spatial neighborhood of the pixel, and
determining, for the pixel, whether the radiance values for the pixel and the pixels in the spatial neighborhood indicate that a sudden illumination change has occurred in a scene depicted by the sequence of video frames; and
upon determining that a majority of pixels in the first set of pixels indicate that a sudden illumination change has occurred, reclassifying one or more pixels in the first set of pixels as depicting a background of the scene in the current frame, wherein the reclassified pixels were classified as depicting scene background in one or more frames prior to the current frame.
20. The system of claim 19 , wherein the operation further comprises:
updating a set of pixels in a background image of the scene corresponding to the pixels in the first set of pixels.
21. The system of claim 19 , wherein the spatial neighborhood for a given pixel comprises a 3×3 grid of pixels radiance values with a radiance value given pixel in the center of the 3×3 grid.
22. The system of claim 19 , wherein determining, for the pixel, whether the radiance values for the pixel and the pixels in the spatial neighborhood indicate that a sudden illumination change has occurred, comprises:
identifying, in a background image, a pixel corresponding to the pixel in the current video frame;
determining an intensity difference between the corresponding pixel and each pixel in a spatial neighborhood of the corresponding pixel;
determining an intensity difference between the pixel and each pixel in the spatial neighborhood of the pixel in the current video frame;
normalizing the intensity differences determined between the pixel and each pixel in the spatial neighborhood of the pixel in the current video frame;
determining an absolute difference between each respective pixel in the spatial neighborhood of the pixel in the current video frame and each respective pixel in the spatial neighborhood of the corresponding pixel;
for each pixel in the spatial neighborhood of the pixel in the current video frame, determining whether the absolute difference is below a first specified threshold, and
if so, incrementing a counter representing how many pixels in the spatial neighborhood are below the first specified threshold;
upon determining the counter exceeds a second threshold, setting a status of the pixel as being an SIC candidate pixel, indicating that the pixel was classified as depicting a scene foreground object due to a sudden illumination change in the scene depicted by the sequence of video frames.
23. The system of claim 22 , wherein the operation further comprises:
upon determining a count of pixels classified as depicting the foreground object which have a status of SIC candidate pixel is below a third threshold, setting a status of the foreground object as being a true foreground object, the status indicating that a sudden illumination change has not occurred for the pixels classified as depicting the foreground in the scene depicted by the sequence of video frames; and
upon determining a count of pixels classified as depicting the foreground object which have a status of SIC candidate pixel is above a fourth threshold, setting a status of the foreground object blob as being an SIC foreground object, the status indicating that a sudden illumination change has occurred for that foreground object in the scene in the current video frame.
24. The system of claim 22 , wherein normalizing the intensity differences comprises, dividing each of the intensity differences determined between the pixel and each pixel in the spatial neighborhood of the pixel in the current video frame using a ratio of the radiance value of the pixel in the current video frame and a radiance value of the corresponding pixel in the background image.
25. The system of claim 22 , wherein the first specified threshold is determined using a Gaussian distribution determined using the pixel radiance value as a mean and an expected camera noise at the pixel radiance value as a variance.