Background-foreground module for video analysis system
View Patent ↗Embodiments of the present invention provide a method and a module for identifying a background of a scene depicted in an acquired stream of video frames that may be used by a video-analysis system. For each pixel or block of pixels in an acquired video frame a comparison measure is determined. The comparison measure depends on difference of color values exhibited in the acquired video frame and in a background image respectively by the pixel or block of pixels and a corresponding pixel and block of pixels in the background image. To determine the comparison measure, the resulting difference is considered in relation to a range of possible color values. If the comparison measure is above a dynamically adjusted threshold, the pixel or the block of pixels is classified as a part of the background of the scene.
1. A computer-implemented method for analyzing a scene depicted in a sequence of video frames, the method comprising:
receiving a raw video frame of the sequence of video frames; and
for each of a plurality of pixels of the raw video frame, by operation of one or more computer processors:
calculating a comparison measure between color-characteristics of a pixel in the raw video frame and color-characteristics of a corresponding pixel in a background image, wherein the comparison measure is defined in relation to a range of possible values of at least one color-characteristic of the pixel;
upon determining the comparison measure of the pixel is within a specified threshold, classifying the pixel of the raw frame as depicting a portion of a background of the scene; and
upon determining the comparison measure is outside the specified threshold, classifying the pixel of the raw frame as depicting a portion of a foreground of the scene.
2. The computer-implemented method of claim 1 , wherein the specified threshold is selected based on a set of predefined brightness ranges for the at least one color-characteristic of the pixel, wherein the predefined brightness ranges include a range defining a dark pixel, a range defining a medium pixel, and a range defining a bright pixel.
3. The computer-implemented method of claim 1 , wherein the calculating the comparison measure of the pixel in the raw video frame comprises:
for each of a plurality of color-characteristics of the pixel:
calculating a measure of closeness between a value of a color-characteristic of the pixel and a value of the color-characteristic of the corresponding pixel in the background image using a range of possible values of the color-characteristic.
4. The computer-implemented method of claim 3 , wherein color-characteristics of the pixel and of the corresponding pixel in the background image are selected from a group consisting of a Red (R), Green (G), and Blue (B) color-channels.
5. The computer-implemented method of claim 3 , wherein calculating the measure of closeness between color-characteristics of the pixel in the raw video frame and color-characteristics of the corresponding pixel in the background image, further comprises:
determining a first value based on a range of possible values of a first color-channel and a difference between values of the first color-channel of the pixel and of the corresponding pixel in the background image; and
determining a second value based on a range of possible values of a second color-channel and a difference between values of the second color-channel of the pixel and of the corresponding pixel in the background image, wherein the first color-channel and the second color-channel are different color-channels.
6. The computer-implemented method of claim 5 , wherein the calculating a measure of closeness between color-characteristics of the pixel in the raw video frame and color-characteristics of the corresponding pixel in the background image, further comprises:
calculating an average of the first value and the second value.
7. The computer-implemented method of claim 5 , wherein the calculating a measure of closeness between color-characteristics of the pixel in the raw video frame and color-characteristics of the corresponding pixel in the background image, further comprises:
determining a third value based on a range of possible values of a third color-channel and a difference between values of the third color-channel of the corresponding pixel in the background image, wherein the third color-channel is a different color-channel than the first color-channel and the second color-channel;
comparing the first value, the second value, and the third value to determine a nosiest color-channel of the pixel; and
calculating the comparison measure by calculating an average of two values selected from a group consisting of the first value, the second value, and the third value, wherein a value corresponding to the noisiest color-channel is disregarded, wherein the noisiest color-channel is a color-channel having a smallest value of the first value, second value, and the third value.
8. The computer-implemented method of claim 1 , wherein the background image includes a short-term background image and a long-term background image.
9. The computer-implemented method of claim 1 , further comprising, training the background image using a set of consecutive video frames of the sequence of video frames.
10. The computer-implemented method of claim 1 , further comprising:
updating the background image using color-characteristic values of each pixel of the raw video frame classified as depicting a portion of the background of the scene.
11. The computer-implemented method of claim 1 , further comprising:
for at least a first pixel classified as depicting a portion of a foreground of the scene, determining that the first pixel was classified as depicting a portion of the foreground of the scene within a specified marginal magnitude of the threshold;
calculating an average set of pixel color-characteristic values from a set of pixels neighboring the first pixel;
calculating a comparison measure for the average set of pixel color-characteristic values and color-characteristics of a corresponding pixel in a background image; and
upon determining the comparison measure for the average set of pixel color-characteristic values is within the specified threshold for classing a pixel as depicting scene background, re-classifying the first pixel of the raw frame as depicting a portion of a background of the scene.
12. A non-transitory computer-readable storage medium containing a program which, when executed by a processor, performs an operation for analyzing a scene depicted in a sequence of video frames, the operation comprising:
receiving a raw video frame of the sequence of video frames; and
for each of a plurality of pixels of the raw video frame:
calculating a comparison measure between color-characteristics of a pixel in the raw video frame and color-characteristics of a corresponding pixel in a background image, wherein the comparison measure is defined in relation to a range of possible values of at least one color-characteristic of the pixel,
upon determining the comparison measure of the pixel is within a specified threshold, classifying the pixel of the raw frame as depicting a portion of a background of the scene, and
upon determining the comparison measure is outside the specified threshold, classifying the pixel of the raw frame as depicting a portion of a foreground of the scene.
13. The non-transitory computer-readable storage medium of claim 12 , wherein the specified threshold is selected based on a set of predefined brightness-ranges for the at least one color-characteristic of the pixel, wherein the predefined brightness ranges include a range defining a dark pixel, a range defining a medium pixel, and a range defining a bright pixel.
14. The non-transitory computer-readable storage medium of claim 12 , wherein the calculating the comparison measure of the pixel in the raw video frame comprises:
for each of a plurality of color-characteristics of the pixel:
calculating a measure of closeness between a value of a color-characteristic of the pixel and a value of the color-characteristic of the corresponding pixel in the background image using a range of possible values of the color-characteristic.
15. The non-transitory computer-readable storage medium of claim 14 , wherein calculating the measure of closeness between color-characteristics of the pixel in the raw video frame and color-characteristics of the corresponding pixel in the background image, further comprises:
determining a first value based on a range of possible values of a first color-channel and a difference between values of the first color-channel of the pixel and of the corresponding pixel in the background image; and
determining a second value based on a range of possible values of a second color-channel and a difference between values of the second color-channel of the pixel and of the corresponding pixel in the background image, wherein the first color-channel and the second color-channel are different color-channels.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the calculating a measure of closeness between color-characteristics of the pixel in the raw video frame and color-characteristics of the corresponding pixel in the background image, further comprises:
calculating an average of the first value and the second value.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the calculating a measure of closeness between color-characteristics of the pixel in the raw video frame and color-characteristics of the corresponding pixel in the background image, further comprises:
determining a third value based on a range of possible values of a third color-channel and a difference between values of the third color-channel of the corresponding pixel in the background image, wherein the third color-channel is a different color-channel than the first color-channel and the second color-channel;
comparing the first value, the second value, and the third value to determine a nosiest color-channel of the pixel; and
calculating the comparison measure by calculating an average of two values selected from a group consisting of the first value, the second value, and the third value, wherein a value corresponding to the noisiest color-channel is disregarded.
18. The non-transitory computer-readable storage medium of claim 12 , wherein the operation further comprises, training the background image using a set of consecutive video frames of the sequence of video frames.
19. The non-transitory computer-readable storage medium of claim 12 , wherein the operation further comprises updating the background image using color-characteristic values of each pixel of the raw video frame classified as depicting a portion of the background of the scene.
20. The non-transitory computer-readable storage medium of claim 12 , wherein the operation further comprises:
for at least a first pixel classified as depicting a portion of a foreground of the scene, determining that the first pixel was classified as depicting a portion of the foreground of the scene within a specified marginal magnitude of the threshold;
calculating an average set of pixel color-characteristic values from a set of pixels neighboring the first pixel;
calculating a comparison measure for the average set of pixel color-characteristic values and color-characteristics of a corresponding pixel in a background image; and
upon determining the comparison measure for the average set of pixel color-characteristic values is within the specified threshold for classing a pixel as depicting scene background, re-classifying the first pixel of the raw frame as depicting a portion of a background of the scene.
21. A system comprising,
a video input source configured to provide a sequence of video frames, each depicting a scene;
a processor; and
a memory containing a video analysis application, which when executed by the processor is configured to perform an operation for analyzing the scene depicted in the sequence of video frames, the operation comprising:
receiving a raw video frame from the video input source, and
for each of a plurality of pixels of the raw video frame:
calculating a comparison measure between color-characteristics of a pixel in the raw video frame and color-characteristics of a corresponding pixel in a background image, wherein the comparison measure is defined in relation to a range of possible values of at least one color-characteristic of the pixel,
upon determining the comparison measure of the pixel is within a specified threshold, classifying the pixel of the raw frame as depicting a portion of a background of the scene, and
upon determining the comparison measure is outside the specified threshold, classifying the pixel of the raw frame as depicting a portion of a foreground of the scene.
22. The system of claim 21 , wherein the specified threshold is selected based on a set of predefined brightness ranges for the at least one color-characteristic of the pixel, wherein the predefined brightness ranges include a range defining a dark pixel, a range defining a medium pixel, and a range defining a bright pixel.
23. The system of claim 21 , wherein the operation further comprises, training the background image using a set of consecutive video frames of the sequence of video frames.
24. The system of claim 21 , wherein the operation further comprises:
updating the background image using color-characteristic values of each pixel of the raw video frame classified as depicting a portion of the background of the scene.
25. The system of claim 21 , wherein the operation further comprises:
for at least a first pixel classified as depicting a portion of a foreground of the scene, determining that the first pixel was classified as depicting a portion of the foreground of the scene within a specified marginal magnitude of the threshold;
calculating an average set of pixel color-characteristic values from a set of pixels neighboring the first pixel;
calculating a comparison measure for the average set of pixel color-characteristic values and color-characteristics of a corresponding pixel in a background image; and
upon determining the comparison measure for the average set of pixel color-characteristic values is within the specified threshold for classing a pixel as depicting scene background, re-classifying the first pixel of the raw frame as depicting a portion of a background of the scene.