Digital video content fingerprinting based on scale invariant interest region detection with an array of anisotropic filters
View Patent ↗Video sequence processing is described with various filtering rules applied to extract dominant features for content based video sequence identification. Active regions are determined in video frames of a video sequence. Video frames are selected in response to temporal statistical characteristics of the determined active regions. A two pass analysis is used to detect a set of initial interest points and interest regions in the selected video frames to reduce the effective area of images that are refined by complex filters that provide accurate region characterizations resistant to image distortion for identification of the video frames in the video sequence. Extracted features and descriptors are robust with respect to image scaling, aspect ratio change, rotation, camera viewpoint change, illumination and contrast change, video compression/decompression artifacts and noise. Compact, representative signatures are generated for video sequences to provide effective query video matching and retrieval in a large video database.
1. A method of image scale-space analysis that applies an array of anisotropic filters for interest region determination, the method comprising:
forming an array of elliptic-shaped anisotropic second order partial derivative Gaussian filters with a first scale parameter in the x direction (s x ) and a second scale parameter in the y direction (s y ) as scale parameters (s x , s y );
forming an array of rectangular-shaped anisotropic second order partial derivative Gaussian filters with finite spatial support according to the (s x , s y ) scale parameters to approximate the elliptic-shaped anisotropic second order partial derivative Gaussian filters;
analyzing scale space for a selected image frame by the array of rectangular-shaped anisotropic second order partial derivative Gaussian filters of finite spatial support to determine at least one interest point corresponding to different (s x , s y ) scale parameters, wherein an interest point identifies a location of a distinct image region in the selected image frame; and
convolving a region of pixels in a surrounding neighborhood of the determined at least one interest point with the array of rectangular anisotropic second order partial derivative Gaussian filters to produce at least one interest point result that is resistant to image distortion.
2. The method of claim 1 further comprising:
computing a Hessian determinant and a Hessian determinant response image for each anisotropic second order partial derivative Gaussian filter with scale parameters (s x , s y ).
3. The method of claim 2 further comprising:
determining a set of local maxima in contiguous 4-dimensional scale-space volumes of Hessian determinant response images, wherein the set of local maxima is further sorted in magnitude and a new set of local maxima is selected that represents a set of non-interpolated interest points.
4. The method of claim 2 further comprising:
interpolating a set of local maxima in a combined spatial-scale 4-dimensional pixel volume of Hessian determinant response images, constructed for a j by j spatial neighborhood and a k by k scale neighborhood of each Hessian determinant response image, to produce a spatial position and (s x , s y ) scale parameters of at least one refined interest point.
5. The method of claim 4 , wherein the at least one refined interest point is described with a 5-tuple (s x , s y , x, y, peak polarity).
6. The method of claim 1 , wherein the scale parameters (s x , s y ) are directly proportional to blurring parameters (σ x , σ y ) used in said Gaussian filters.
7. The method of claim 1 , wherein the at least one interest point result includes a location coordinate x, y and scale parameters s x , s y for the determined at least one interest point.
8. The method of claim 1 further comprising:
establishing a rectangular array of pixels centered at the determined at least one interest point in the selected image frame and sub-divided into equal sized pixel arrays, wherein each pixel array is comprised of re-sampled pixels;
computing gradient functions for each pixel array, wherein each computed gradient function produces a multi-bit value representing a dimension of the rectangular array; and
combining the produced multi-bit values for each pixel array to form a multi-dimensional descriptor.
9. The method of claim 8 , wherein the rectangular array is sized in proportion to the (s x , s y ) scale parameters.
10. The method of claim 8 further comprising:
reducing the number of bits per dimension in each gradient multi-bit value to form a multi-dimensional signature, wherein the formed multi-dimensional signature is a compact representation of the determined at least one interest point.
11. A method of image scale-space analysis that applies an array of anisotropic filters for interest region determination, the method comprising:
forming a filter array comprised of elliptic-shaped anisotropic second order partial derivative Gaussian filters with scale parameters (s x , s y ), s x a scale parameter in the x direction and s y a scale parameter in the y direction;
applying each elliptic-shaped anisotropic second order partial derivative Gaussian filter of the filter array to an image frame to generate second order partial derivative images;
computing a determinant of a Hessian matrix of the second order partial derivative images at selected interest points with associated (s x , s y ) scale parameters to form determinant images, wherein an interest point identifies a location of a distinct image region in the image frame; and
selecting a set of local maxima interest points with associated vectors (s x , s y , x, y, peak value) from the determinant images for signature generation, wherein the selected set of local maxima interest points are resistant to image distortion.
12. The method of claim 11 further comprising:
computing local maxima values for groups of pixels selected from the determinant images tiled into a four dimensional volume comprising k by k spatial neighborhoods and an s x by s y scale neighborhoods for the associated (s x , s y ) scale parameters.
13. The method of claim 12 further comprising:
sorting the computed local maxima values according to their magnitudes and selecting local maxima values that exceed a threshold as a set of refined interest points.
14. The method of claim 13 further comprising:
computing a second order Taylor series approximation based on values in a scale-space neighborhood of each refined interest point to determine the set of local maxima interest points.
15. The method of claim 11 , wherein the filter array of elliptic-shaped anisotropic second order partial derivative Gaussian filters is formed across a spectrum of s x values and s y values, with filters having the same s x and s y values are isotropic.
16. The method of claim 11 , wherein the second order partial derivative images are computed by a convolution of an image interest point with a second order partial derivative of a spatial point and scale pair (σ x , σ y ) used in said Gaussian filters.
17. The method of claim 16 , wherein the scale parameters (s x , s y ) are directly proportional to the scale pair (σ x , σ y ).
18. The method of claim 11 , wherein the signature generation comprises:
generating a multi-dimensional descriptor in a rectangular region of pixels centered on an interest point selected from the set of local maxima interest points, the rectangular region having a spatial extent proportional to the s x scale parameter and the s y scale parameter and partitioned into sub-regions representing dimensions of the descriptor;
computing gradient functions for each sub-region providing multiple bit results per dimension of the multi-dimensional descriptor; and
generating a signature from the multi-dimensional descriptor by reducing the number of bits pre dimension, wherein the generated signature is a compact representation of the selected interest point.
19. A non-transitory computer readable medium storing a computer program which causes a computer system to perform a method of image scale-space analysis that applies an array of anisotropic filters for interest region determination, the method comprising:
forming an array of elliptic-shaped anisotropic second order partial derivative Gaussian filters with a first scale parameter in the x direction (s x ) and a second scale parameter in the y direction (s y ) as scale parameters (s x , s y );
forming an array of rectangular-shaped anisotropic second order partial derivative Gaussian filters with finite spatial support according to the (s x , s y ) scale parameters to approximate the elliptic-shaped anisotropic second order partial derivative Gaussian filters;
analyzing scale space for a selected image frame by the array of rectangular-shaped anisotropic second order partial derivative Gaussian filters of finite spatial support to determine at least one interest point corresponding to different (s x , s y ) scale parameters, wherein an interest point identifies a location of a distinct image region in the selected image frame; and
convolving a region of pixels in a surrounding neighborhood of the determined at least one interest point with the array of rectangular anisotropic second order partial derivative Gaussian filters to produce at least one interest point result that is resistant to image distortion.
20. The non-transitory computer readable medium of claim 19 further comprising:
applying bi-level filters of various spatial support on the selected image frame to produce multiple filter response images with the same spatial support as an original image frame;
constructing a set of contiguous rectangular spatial areas on filter response images derived for the selected image frame;
constructing a corresponding set of 3-dimensional scale-space pixel volumes for each of the contiguous rectangular spatial areas; and
determining a set of local filter response maxima at spatial-scale 3-dimensional pixel volumes, wherein the set of local filter response maxima with their spatial (x, y) coordinates represents initial interest points of a first pass of selected image frame analysis.