Media fingerprinting and identification system
The overall architecture and details of a scalable video fingerprinting and identification system that is robust with respect to many classes of video distortions is described. In this system, a fingerprint for a piece of multimedia content is composed of a number of compact signatures, along with traversal hash signatures and associated metadata. Numerical descriptors are generated for features found in a multimedia clip, signatures are generated from these descriptors, and a reference signature database is constructed from these signatures. Query signatures are also generated for a query multimedia clip. These query signatures are searched against the reference database using a fast similarity search procedure, to produce a candidate list of matching signatures. This candidate list is further analyzed to find the most likely reference matches. Signature correlation is performed between the likely reference matches and the query clip to improve detection accuracy.
1. A system for content based video sequence identification comprising: a processing unit; and
a non-transitory computer-readable medium having stored therein instructions that are executable to cause the processing unit to perform functions comprising:
applying a bi-level filter to images in a first pass analysis to detect a set of initial interest points in a plurality of selected video frames, wherein the first pass analysis reduces an effective area of the images in each selected video frame to multiple smaller images, and
applying an array of anisotropic filters to regions of pixels around each initial interest point of the set of initial interest points in a second pass analysis to refine a spatial position for each initial interest point and determine a first scale parameter in an x direction (S x ) and a second scale parameter in a y direction (S y ), wherein the S x and the S y scale parameters are separately varied to provide accurate region characterizations that are resistant to image distortion for identification of the plurality of selected video frames in a video sequence.
2. The system of claim 1 , wherein the array of anisotropic filters is an array of sampled elliptic-shaped anisotropic filters.
3. The system of claim 1 , wherein the functions further comprise:
applying bi-level symmetric filters with multiple different spatial support on a selected video frame to produce multiple filter response images with the same size selected video frame;
constructing a set of contiguous rectangular spatial areas on the multiple filter response images derived for the selected video frame;
constructing a corresponding set of 3-dimensional scale space pixel volumes for each of the contiguous rectangular spatial areas;
determining a set of local filter response maxima at spatial scale 3-dimensional pixel volumes;
sorting local filter response maxima at spatial-scale 3-dimensional pixel volumes; and
selecting a set of local filter response maxima with their spatial (x, y) coordinates to represents the initial interest points of the first pass analysis of the selected video frame.
4. The system of claim 1 , wherein the functions further comprise convolving Laplacian of Gaussian second order partial derivative bi-level filters of various sizes with one of the selected video frames to form bi-level filter response images for a specified set of scalar scale values.
5. The system of claim 1 , wherein the functions further comprise computing a set of Hessian determinant response images for each interest region formed around an initial interest point to determine a (s x , s y ) scale pair for each initial interest point, wherein each initial interest point is detected in the first pass analysis.
6. The system of claim 1 , wherein the functions further comprise forming a Hessian matrix and a Hessian determinant response image with pixels representing a determinant of the Hessian matrix for each anisotropic filter from the array of anisotropic filters with spatial support corresponding to (s x , s y ) scale values.
7. The system of claim 1 , wherein the functions further comprise computing a set of Hessian determinant response images for each interest region formed around an initial interest point to refine an (x, y) position of the initial interest point.
8. The system of claim 1 , wherein the bi-level filter comprises a Laplacian of Gaussian second order partial derivative bi-level filter.
9. A non-transitory computer-readable medium having stored therein instructions that are executable to cause a processing unit to perform functions comprising:
applying a bi-level filter to images in a first pass analysis to detect a set of initial interest points in a plurality of selected video frames, wherein the first pass analysis reduces an effective area of the images in each selected video frame to multiple smaller images; and
applying an array of anisotropic filters to regions of pixels around each initial interest point of the set of initial interest points in a second pass analysis to refine a spatial position for each initial interest point and determine a first scale parameter in an x direction (s x ) and a second scale parameter in a y direction (s y ), wherein the s x and the s y scale parameters are separately varied to provide accurate region characterizations that are resistant to image distortion for identification of the plurality of selected video frames in a video sequence.
10. The non-transitory computer-readable medium of claim 9 , wherein the array of anisotropic filters is an array of sampled elliptic-shaped anisotropic filters.
11. The non-transitory computer-readable medium of claim 9 , wherein the functions further comprise:
applying bi-level symmetric filters with multiple different spatial support on a selected video frame to produce multiple filter response images with the same size selected video frame;
constructing a set of contiguous rectangular spatial areas on the multiple filter response images derived for the selected video frame;
constructing a corresponding set of 3-dimensional scale space pixel volumes for each of the contiguous rectangular spatial areas;
determining a set of local filter response maxima at spatial scale 3-dimensional pixel volumes;
sorting local filter response maxima at spatial-scale 3-dimensional pixel volumes; and
selecting a set of local filter response maxima with their spatial (x, y) coordinates to represents the initial interest points of the first pass analysis of the selected video frame.
12. The non-transitory computer-readable medium of claim 9 , wherein the functions further comprise convolving Laplacian of Gaussian second order partial derivative bi-level filters of various sizes with one of the selected video frames to form bi-level filter response images for a specified set of scalar scale values.
13. The non-transitory computer-readable medium of claim 9 , wherein the functions further comprise computing a set of Hessian determinant response images for each interest region formed around an initial interest point to determine a (s x , s y ) scale pair for each initial interest point, wherein each initial interest point is detected in the first pass analysis.
14. The non-transitory computer-readable medium of claim 9 , wherein the functions further comprise forming a Hessian matrix and a Hessian determinant response image with pixels representing a determinant of the Hessian matrix for each anisotropic filter from the array of anisotropic filters with spatial support corresponding to (s x , s y ) scale values.
15. The non-transitory computer-readable medium of claim 9 , wherein the functions further comprise computing a set of Hessian determinant response images for each interest region formed around an initial interest point to refine an (x, y) position of the initial interest point.
16. The non-transitory computer-readable medium of claim 9 , wherein the bi-level filter comprises a Laplacian of Gaussian second order partial derivative bi-level filter.
17. A system for content based video sequence identification comprising:
a processing unit; and
a non-transitory computer-readable medium having stored therein instructions that are executable to cause the processing unit to perform functions comprising:
applying a bi-level filter in a first pass analysis to detect a set of initial interest points in selected video frames, wherein the first pass analysis reduces an
effective area of images in each selected video frame to multiple smaller images;
applying an array of anisotropic filters to regions of pixels around the set of initial interest points in a second pass analysis to form a 4-dimensional (4D) space of determinant images with coordinate (x, y, s x , s y ) values; and interpolating the determinant images to identify refined interest points
with coordinate (x, y, s x , s y ) values that provide accurate region characterizations that are resistant to image distortion for identification of the video frames in the video sequence.
18. The system of claim 17 , wherein the functions further comprise identifying a refined interest point by a first scale parameter in an x direction (s x ) and a second scale parameter in a y direction (s y ), wherein the s x and s y scale parameters define a spatial extent in the x direction and a spatial extent in the y direction of an elliptic-shaped image for each applied anisotropic filter.
19. The system of claim 18 , wherein the functions further comprise generating an interest point descriptor for a rectangular region around the identified refined interest point with rectangular vertices that are proportional to the s x and s y values of the identified refined interest point having the coordinate (x, y, s x , s y ) values.
20. The system of claim 18 , wherein the functions further comprise:
generating a k by k grid in an Ns x by Ms x region centered around the identified refined interest point and a j by j re-sampled sub region containing j 2 interpolated pixels for each cell of the k by k grid, wherein N and M are multiplication factors which determine a neighborhood size around the refined interest point;
generating a horizontal gradient Gx and a vertical gradient Gy based on a partial derivative of each pixel in the j by j re-sampled sub-region; and
generating a plurality of computed gradient values for each sub-region to be concatenated providing a descriptor for the identified refined interest point.