Multi-source, multi-scale counting in dense crowd images
View Patent ↗A method for counting individuals in an image containing a dense, uniform or non-uniform crowd. The current invention leverages multiple sources of information to compute an estimate of the number of individuals present in a dense crowd visible in a single image. This approach relies on multiple sources, such as low confidence head detections, repetition of texture elements (using SIFT), and frequency-domain analysis to estimate counts, along with confidence associated with observing individuals in an image region. Additionally, a global consistency constraint can be employed on counts using Markov Random Field. This caters for disparity in counts in local neighborhoods and across scales. The methodology was tested on a new dataset of fifty (50) crowd images containing over 64,000 annotated humans, with the head counts ranging from 94 to 4,543. Efficient and accurate results were attained.
1. One or more non-transitory tangible computer-readable media having computer-executable instructions for performing a method by running a software program on a computer, the computer operating under an operating system, the method including issuing instructions from the software program to count or estimate a number of individuals in an image of a dense, uniform or non-uniform crowd, the instructions comprising:
receiving and displaying said image on an electronic display device, said image including a plurality of individuals in said dense, uniform or non-uniform crowd;
dividing said image into a series of patches, wherein a patch of said series of patches includes a pattern of peaks;
detecting repetitions of said pattern, wherein a periodic occurrence of said peaks in said repetitions of said pattern indicate a first estimated count of said individuals in said patch based on a first granularity of said patch;
detecting and localizing objects associated with said plurality of individuals in said image via head detection in order to automatically identify a second estimated count of said individuals in said patch based on a second granularity of said patch, wherein said second granularity of said patch is larger than said first granularity of said patch;
receiving a database including a plurality of descriptions of local features that may or may not be associated with said individuals being counted in said patch;
detecting said local features in said patch, said local features that are associated with said individuals indicating a third estimated count of said individuals in said patch based on a third granularity of said patch, wherein said third granularity of said patch is smaller than said first granularity of said patch;
inputting said first estimated count, said second estimated count, and said third estimated into a multidimensional vector;
automatically sweeping each layer of said multidimensional vector to determine beliefs at said each layer of said multidimensional vector, resulting in a patch count for said patch, wherein an evaluation of data term or unary cost for said patch at a layer of said multidimensional vector is independent of layers above or below said layer;
repeating the foregoing steps with each patch of said series of patches; and
automatically computing a resulting estimated count of said image based on said beliefs of said each patch.
2. One or more non-transitory tangible computer-readable media, as in claim 1 , further comprising:
applying a filter corresponding to heads of said plurality of individuals during said step of detecting and localizing said objects.
3. One or more non-transitory tangible computer-readable media, as in claim 1 , further comprising:
incorporating scale and confidence into said first estimated count, said second estimated count, and said third estimated count for minimizing threshold of detection as a result of said image being occluded,
wherein said first, second, and third estimated counts include said scales and confidences.
4. One or more non-transitory tangible computer-readable media, as in claim 1 , further comprising:
computing a gradient image, ∇(P), of said image; and
applying a low-pass filter, f(ξ)>f(ξ o )=0, to remove high frequency content prior to calculating said first estimated count.
5. One or more non-transitory tangible computer-readable media, as in claim 1 , further comprising:
reconstructing said image via inverse Fourier transform prior to calculating said first estimated count; and
detecting a number of local maximas in said reconstructed image after alignment and non-maximal suppression, said number of local maximas being an estimate for said first estimated count of said number of individuals in said image.
6. One or more non-transitory tangible computer-readable media, as in claim 5 , further comprising:
determining entropy, mean, variance, skewness, and kurtosis of said reconstructed image and of a difference image after detecting said peaks but prior to calculating said first estimated count, said difference image being an absolute difference between said reconstructed image and a gradient image of said image; and
normalizing said first estimated count for a size of said patch.
7. One or more non-transitory tangible computer-readable media, as in claim 1 , further comprising:
said step of sweeping each layer performed by conducting four (4) sweeps at a bottom layer of said multidimensional vector to determine said beliefs for intermediate nodes of said multidimensional vector above said bottom layer, followed by conducting four (4) sweeps at a second layer of said multidimensional vector above said bottom layer to determine said beliefs at said second layer, and
repeating the foregoing step but beginning with sweeping said second layer, followed by sweeping said bottom layer.
8. One or more non-transitory tangible computer-readable media, as in claim 1 , further comprising:
applying a smoothness constraint to a spatial neighborhood of said patch to improve accuracy of said patch count using a Markov random field.
9. One or more non-transitory tangible computer-readable media, as in claim 1 , further comprising:
said descriptions including local features describing external objects that are not associated with said individuals being counted in said patch; and
automatically discarding an external object included in said database from said third estimated count of said individuals in said patch, said third estimated count including said local features that are associated with said individuals less said local features that are not associated with said individuals.
10. A computer-implemented method of counting or estimating a number of individuals in an image of a dense, uniform or non-uniform crowd, comprising:
receiving and displaying said image on an electronic display device, said image including a plurality of individuals in said dense, uniform or non-uniform crowd;
dividing said image into a series of patches, wherein a patch of said series of patches includes a pattern of peaks;
detecting repetitions of said pattern, wherein a periodic occurrence of said peaks in said repetitions of said pattern indicate a first estimated count of said individuals in said patch based on a first granularity of said patch;
detecting and localizing objects associated with said plurality of individuals in said image via head detection in order to automatically identify a second estimated count of said individuals in said patch based on a second granularity of said patch, wherein said second granularity of said patch is larger than said first granularity of said patch;
receiving a database including a plurality of descriptions of local features that may or may not be associated with said individuals being counted in said patch;
detecting said local features in said patch, said local features that are associated with said individuals indicating a third estimated count of said individuals in said patch based on a third granularity of said patch, wherein said third granularity of said patch is smaller than said first granularity of said patch;
inputting said first estimated count, said second estimated count, and said third estimated into a multidimensional vector;
automatically sweeping each layer of said multidimensional vector to determine beliefs at said each layer of said multidimensional vector, resulting in a patch count for said patch, wherein an evaluation of data term or unary cost for said patch at a layer of said multidimensional vector is independent of layers above or below said layer;
repeating the foregoing steps with each patch of said series of patches; and
automatically computing a resulting estimated count of said image based on said beliefs of said each patch.
11. A computer-implemented method as in claim 10 , further comprising:
applying a filter corresponding to heads of said plurality of individuals during said step of detecting and localizing said objects.
12. A computer-implemented method as in claim 10 , further comprising:
incorporating scale and confidence into said first estimated count, said second estimated count, and said third estimated count for minimizing threshold of detection as a result of said image being occluded,
wherein said first, second, and third estimated counts include said scales and confidences.
13. A computer-implemented method as in claim 10 , further comprising:
computing a gradient image, ∇(P), of said image; and
applying a low-pass filter, f(ξ) >f(ξ o )=0, to remove high frequency content prior to calculating said first estimated count.
14. A computer-implemented method as in claim 10 , further comprising:
reconstructing said image via inverse Fourier transform prior to calculating said first estimated count; and
detecting a number of local maximas in said reconstructed image after alignment and non-maximal suppression, said number of local maximas being an estimate for said first estimated count of said number of individuals in said image.
15. A computer-implemented method as in claim 14 , further comprising:
determining entropy, mean, variance, skewness, and kurtosis of said reconstructed image and of a difference image after detecting said peaks but prior to calculating said first estimated count, said difference image being an absolute difference between said reconstructed image and a gradient image of said image; and
normalizing said first estimated count for a size of said patch.
16. A computer-implemented method as in claim 10 , further comprising:
said step of sweeping each layer performed by conducting four (4) sweeps at a bottom layer of said multidimensional vector to determine said beliefs for intermediate nodes of said multidimensional vector above said bottom layer, followed by conducting four (4) sweeps at a second layer of said multidimensional vector above said bottom layer to determine said beliefs at said second layer, and
repeating the foregoing step but beginning with sweeping said second layer, followed by sweeping said bottom layer.
17. A computer-implemented method as in claim 10 , further comprising:
applying a smoothness constraint to a spatial neighborhood of said patch to improve accuracy of said patch count using a Markov random field.
18. A computer-implemented method as in claim 10 , further comprising:
said descriptions including local features describing external objects that are not associated with said individuals being counted in said patch; and
automatically discarding an external object included in said database from said third estimated count of said individuals in said patch, said third estimated count including said local features that are associated with said individuals less said local features that are not associated with said individuals.
19. One or more non-transitory tangible computer-readable media having computer-executable instructions for performing a method by running a software program on a computer, the computer operating under an operating system, the method including issuing instructions from the software program to count or estimate a number of individuals in an image of a dense, non-uniform crowd, the instructions comprising:
receiving and displaying said image on an electronic display device, said image including a plurality of individuals in said dense, uniform or non-uniform crowd;
dividing said image into a series of patches, wherein a patch of said series of patches includes a pattern of peaks;
computing a gradient image, ∇(P), of said image;
applying a low-pass filter, f(ξ)>f(ξ o )=0, to remove high frequency content;
reconstructing said image via inverse Fourier transform prior to calculating said first estimated count; and
detecting a number of local maximas in said reconstructed image after alignment and non-maximal suppression;
detecting repetitions of said pattern,
determining entropy, mean, variance, skewness, and kurtosis of said reconstructed image and of a difference image after detecting said peaks but prior to calculating said first estimated count, said difference image being an absolute difference between said reconstructed image and a gradient image of said image,
wherein a periodic occurrence of said peaks in said repetitions of said pattern indicate a first estimated count of said individuals in said patch based on a first granularity of said patch, said number of local maximas being an estimate for said first estimated count of said number of individuals in said image;
normalizing said first estimated count for a size of said patch;
detecting and localizing objects associated with said plurality of individuals in said image via head detection in order to automatically identify a second estimated count of said individuals in said patch based on a second granularity of said patch, wherein said second granularity of said patch is larger than said first granularity of said patch;
applying a filter corresponding to heads of said plurality of individuals during said step of detecting and localizing said objects;
detecting said local features in said patch, said local features that are associated with said individuals indicating a third estimated count of said individuals in said patch based on a third granularity of said patch wherein said third granularity of said patch is smaller than said first granularity of said patch;
receiving a database including a plurality of descriptions of local features that may or may not be associated with said individuals being counted in said patch, said descriptions including local features describing external objects that are not associated with said individuals being counted in said patch;
automatically discarding an external object included in said database from said third estimated count of said individuals in said patch, said third estimated count including said local features that are associated with said individuals less said local features that are not associated with said individuals;
incorporating scale and confidence into said first estimated count, said second estimated count, and said third estimated count for minimizing threshold of detection as a result of said image being occluded, wherein said first, second, and third estimated counts include said scales and confidences;
inputting said first estimated count, said second estimated count, and said third estimated into a multidimensional vector;
automatically sweeping each layer of said multidimensional vector to determine beliefs at said each layer of said multidimensional vector, resulting in a patch count for said patch, wherein an evaluation of data term or unary cost for said patch at a layer of said multidimensional vector is independent of layers above or below said layer, said step of sweeping each layer performed by conducting four (4) sweeps at a bottom layer of said multidimensional vector to determine said beliefs for intermediate nodes of said multidimensional vector above said bottom layer, followed by conducting four (4) sweeps at a second layer of said multidimensional vector above said bottom layer to determine said beliefs at said second layer;
repeating the foregoing sweeping step but beginning with sweeping said second layer, followed by sweeping said bottom layer;
applying a smoothness constraint to a spatial neighborhood of said patch to improve accuracy of said patch count using a Markov random field;
repeating the foregoing steps with each patch of said series of patches; and
automatically computing a resulting estimated count of said image based on said beliefs of said each patch.