Media fingerprinting and identification system
View Patent ↗The overall architecture and details of a scalable video fingerprinting and identification system that is robust with respect to many classes of video distortions is described. In this system, a fingerprint for a piece of multimedia content is composed of a number of compact signatures, along with traversal hash signatures and associated metadata. Numerical descriptors are generated for features found in a multimedia clip, signatures are generated from these descriptors, and a reference signature database is constructed from these signatures. Query signatures are also generated for a query multimedia clip. These query signatures are searched against the reference database using a fast similarity search procedure, to produce a candidate list of matching signatures. This candidate list is further analyzed to find the most likely reference matches. Signature correlation is performed between the likely reference matches and the query clip to improve detection accuracy.
1. A method of searching reference signatures for a query image, the method comprising:
storing a plurality of N dimension reference signatures together beginning at a starting location in a reference signature database, wherein each reference signature is stored with a link to a corresponding reference image;
generating an N dimension query signature and a traversal hash query signature for a query image; and
searching the reference signature database using the traversal hash query signature as an address into the reference signature database to find similar signatures that are within a specified signature distance R from each other, wherein the similar signatures represent reference images that are similar to the query image.
2. The method of claim 1 further comprising:
quantizing dimensions of an N dimension reference descriptor to form an N dimension reference signature; and
quantizing dimensions of the N dimension reference signature to form a traversal hash reference signature and the traversal hash reference signature points to the starting location in the reference signature database wherein a cluster of related reference signatures are to be stored.
3. The method of claim 1 , wherein the traversal hash query signature is generated based on a quantization of dimensions of an N dimension query descriptor used to form the N dimension query signature.
4. The method of claim 1 further comprising:
organizing information for a current image into a sets of scale (sx, sy) factors and associated region of interest center location (centerx, centery) values, associated reference descriptors for each region of interest, and associated reference signatures and traversal hash reference signatures for each region of interest.
5. The method of claim 1 further comprising:
determining descriptor information related to an evaluation of a descriptor for each region of interest in a current image and according to a minimum threshold of acceptance;
determining signature information related to an evaluation of a signature for each region of interest in the current image and according to a minimum threshold of acceptance; and
determining spatial diversity among scale (sx, sy) factors and center location (centerx, centery) values for each region of interest in the current image and according to a closeness threshold to remove associated signatures that are too close to each other.
6. The method of claim 5 further comprising:
sorting the descriptor information, the signature information, and the spatial diversity for each reference signature and traversal hash reference signature to determine a listing of top T signatures that satisfy the established criteria; and
storing the top T signatures including the associated reference signatures and traversal hash reference signatures along with image identification as a signature record stored in the reference signature database.
7. The method of claim 1 further comprising:
calculating an average distance between a feature's center location (centerx, centery) values and M near features' (centerx, centery) values to indicate a density of features, wherein M is a program specified value; and
removing signatures for features of the M near features that are determined to be too close based on a comparison with a closeness threshold, wherein features that are spread out are preferred.
8. The method of claim 1 , further comprising:
determining an N dimension reference descriptor for a region of interest by partitioning the region of interest into subregions and determining pixel features within each subregion;
determining descriptor information that identifies an N dimension reference descriptor having energy concentrated in a few dimensions of the N dimension reference descriptor that tend to produce false positive matches in search operations; and
removing the N dimension reference descriptor, an associated N dimension reference signature and an associated traversal hash reference signature, that tends to produce false positive matches from a listing of candidate information, wherein a selected number of entries in the listing of candidate information are stored in the reference signature database.
9. The method of claim 1 further comprising:
determining signature information to include a count of a number of asserted bits in each N dimension reference signature of the plurality of N dimension reference signatures; and
removing an N dimension reference signature having a count less than a minimum criteria threshold from a listing of candidate information to be stored in the reference signature database.
10. The method of claim 1 further comprising:
determining an N dimension reference descriptor for a region of interest in an image having an x dimension scale (sx) factor and a y dimension scale (sy) factor that is different from the sx factor; and
quantizing each dimension component of the N dimension reference descriptor according to a quantizing threshold to form an N dimension reference signature for the region of interest.
11. The method of claim 1 further comprising:
creating in an image a grid of Bx by By subregions centered at an interest point of a region of interest having dimensions specified by a scale x (sx) value and scale y (sy) value, wherein each subregion having a grid of Cx by Cy pixel positions;
computing x-gradient values based on pixel intensity differences in an x direction and y-gradient values based on pixel intensity differences in a y direction in each grid of Cx by Cy pixel positions; and
concatenating weighted sum of x-gradient values with weighted sum of y-gradient values for each subregion of the Bx by By subregions to create an N dimension query descriptor.
12. The method of claim 11 , wherein the weighted sum of x-gradient values includes a negative one times a sum of the x-gradient values for x-gradient values that are negative and a sum of the x-gradient values for x-gradient values that are positive and wherein the weighted sum of y-gradient values includes a negative one times a sum of the y-gradient values for y-gradient values that are negative and a sum of the y-gradient values for y-gradient values that are positive.
13. The method of claim 11 , wherein the N dimension query descriptor is created with N=k*Bx*By dimensions, k a number of features evaluated within the region of interest.
14. The method of claim 11 further comprising:
changing the scale sx value and the scale sy value by a multiplicative factor for the grid of Bx by By subregions to generate a second N dimension query descriptor for the same region of interest; and
using a first traversal hash reference signature determined from the N dimension query descriptor and a second traversal hash reference signature determined from the second N dimension query descriptor to improve chances of finding reference images that are similar to the query image.
15. The method of claim 1 , wherein the reference signature database is located on a portable device.
16. A method of searching reference signatures for a query image, the method comprising:
generating a plurality of N dimension query signatures and associated traversal hash query signatures for a query image;
fetching from a reference signature database a set of signature records located at an address set to a selected traversal hash query signature;
computing a bit distance between an N dimension query signature associated with the selected traversal hash query signature and with signatures in the set of signature records; and
storing in a candidate list of matching signatures a signature from the set of signature records as a candidate signature that has the computed bit distance less than a program specified signature distance R.
17. The method of claim 16 , wherein the candidate signature is stored along with an associated reference signature database image identification (ID) and the signature bit distance.
18. The method of claim 17 further comprising:
counting the number of times reference signature database image IDs appears in the candidate list to generate a frequency of occurrence for each reference signature database image in the candidate list;
sorting the candidate list in descending frequency order to generate a sorted frequency list representing the top matches.
19. The method of 16 further comprising:
determining an N dimension query descriptor for a region of interest in the query image, the region of interest having an x dimension scale (sx) factor and a y dimension scale (sy) factor that is different from the sx factor;
combining dimension components of the N dimension query descriptor to generate a combined dimension query descriptor; and
quantizing each combined dimension component of the combined dimension query descriptor according to a traversal hash threshold to form the selected traversal hash query signature.
20. A computer readable non-transitory medium storing a computer program which causes a computer system to perform a method of searching reference signatures for a query image, the method comprising:
storing a plurality of N dimension reference signatures together beginning at a starting location in a reference signature database, wherein each reference signature is stored with a link to a corresponding reference image;
generating an N dimension query signature and a traversal hash query signature for a query image; and
searching the reference signature database using the traversal hash query signature as an address into the reference signature database to find similar signatures that are within a specified signature distance R from each other, wherein the similar signatures represent reference images that are similar to the query image.
21. The computer readable non-transitory medium of claim 20 , wherein the reference signature database is located on a portable device.