Visual saliency estimation for images and video
Methods and apparatus for deriving a saliency measure for images and video are described. In an embodiment, a process includes decomposing, by a processor, an image into elements, wherein elements cluster the image and each element consists of spatially connected pixels. The processor then calculates a first image measure indicative of each element's uniqueness in the image on a per element basis, and calculates a second image measure indicative of each element's spatial distribution in the image on a per element basis. The processor then provides a per element saliency measure by combining the first image measure and the second image measure.
1. A method for deriving a saliency measure for an image of a video frame, comprising:
decomposing, by a processor, the image into elements, wherein elements cluster the image and each element consists of spatially connected pixels;
calculating, by the processor, a first image measure indicative of each element's uniqueness in the image on a per element basis and
a second image measure indicative of each element's spatial distribution in the image on a per element basis, wherein calculating the first and second image measures comprises:
identifying, by the processor, based on a saliency measure of an adjacent video frame image, a background portion of the adjacent video frame image,
computing, by the processor, based on the identified background portion, a global transformation that aligns the video frame image with the adjacent video frame image, and
computing, by the processor, the first and second image measures based on the global transformation; and
providing, by the processor, a per element saliency measure by combining the first image measure and the second image measure.
2. The method of claim 1 , further comprising generating, by the processor, based on the per element saliency measure, a pixel-accurate image saliency map.
3. The method of claim 2 , wherein generating the pixel accurate saliency map comprises utilizing an up-sampling process that is both local and feature sensitive.
4. The method of claim 1 , wherein at least one of the first image measure and the second image measure is calculated based on at least one discriminating feature extracted from the pixels associated with each element.
5. The method of claim 4 , wherein the discriminating feature comprises at least one of a color feature, a texture feature, a disparity feature, and a motion feature.
6. The method of claim 5 , wherein the motion feature is based on an optical flow.
7. The method of claim 1 , wherein decomposing comprises utilizing, by the processor, a modified simple linear iterative clustering (SLIC) superpixels protocol that utilizes K-means clustering in geodesic image distance in CEILab space.
8. The method of claim 1 , wherein at least one of the calculating of the first image measure and the calculating of the second image measure comprises utilizing Gaussian filtering.
9. The method of claim 1 , wherein combining the first image measure and the second image measure comprises:
normalizing uniqueness measures and distribution measures to a range; and
combining each normalized uniqueness measure with a corresponding distribution measure to obtain the per element saliency measure.
10. A non-transitory computer-readable medium storing instructions for deriving a saliency measure for an image of a video frame configured to cause a processor to:
decompose the image into elements, wherein elements cluster the image and each element consists of spatially connected pixels;
calculate a first image measure indicative of each element's uniqueness in the image on a per element basis and
a second image measure indicative of each element's spatial distribution in the image on a per element basis, wherein calculating the first and second image measures comprises:
identifying, based on a saliency measure of an adjacent video frame image, a background portion of the adjacent video frame image,
computing, based on the identified background portion, a global transformation that aligns the video frame image with the adjacent video frame image, and
computing the first and second image measures based on the global transformation; and
provide a per element saliency measure by combining the first image measure and the second image measure.
11. The computer-readable medium of claim 10 , further comprising instructions configured to cause the processor to generate, based on the per element saliency measure, a pixel-accurate image saliency map.
12. The computer-readable medium of claim 11 , wherein the instructions for generating the pixel accurate saliency map further comprise instructions configured to cause the processor to utilize an up-sampling process that is both local and feature sensitive.
13. The computer-readable medium of claim 10 , wherein the instructions for calculating the first image measure or the second image measure further comprise instructions configured to cause the processor to calculate the first image measure or the second image measure based on at least one discriminating feature extracted from the pixels associated with each element.
14. The computer-readable medium of claim 13 , wherein the discriminating feature comprises at least one of a color feature, a texture feature, a disparity feature, and a motion feature.
15. The computer-readable medium of claim 14 , wherein the motion feature is based on an optical flow.
16. The computer-readable medium of claim 10 , wherein the instructions for decomposing further comprise instructions configured to cause the processor to utilize a modified simple linear iterative clustering (SLIC) superpixels protocol that utilizes K-means clustering in geodesic image distance in CEILab space.
17. The computer-readable medium of claim 10 , wherein the instructions for calculating the first and second image measures further comprise instructions configured to cause the processor to utilize Gaussian filtering.
18. The computer-readable medium of claim 10 , wherein the instructions for combining the first and second image measures further comprise instructions configured to cause the processor to:
normalize uniqueness measures and distribution measures to a range; and
combine each normalized uniqueness measure with a corresponding distribution measure to obtain the per element saliency measure.
19. An apparatus for deriving a saliency measure for an image of a video frame, comprising:
a processor;
a communication device operably connected to the processor; and
a storage device operably connected to the processor, the storage device storing non-transitory instructions configured to cause the processor to:
decompose the image into elements, wherein elements cluster the image and each element consists of spatially connected pixels;
calculate a first image measure indicative of each element's uniqueness in the image on a per element basis and
a second image measure indicative of each element's spatial distribution in the image on a per element basis, wherein calculating the first and second image measures comprises:
identifying, based on a saliency measure of an adjacent video frame image, a background portion of the adjacent video frame image,
computing, based on the identified background portion, a global transformation that aligns the video frame image with the adjacent video frame image, and
computing the first and second image measures based on the global transformation; and
provide a per element saliency measure by combining the first image measure and the second image measure.
20. The apparatus of claim 19 , wherein the instructions for calculating at least one of the first image measure and the second image measure stored in the storage device further comprises instructions configured to cause the processor to utilize at least one of a color feature, a texture feature, a disparity feature, and a motion feature.
21. The apparatus of claim 20 , wherein the motion feature is based on an optical flow.