Media fingerprinting and identification system
The overall architecture and details of a scalable video fingerprinting and identification system that is robust with respect to many classes of video distortions is described. In this system, a fingerprint for a piece of multimedia content is composed of a number of compact signatures, along with traversal hash signatures and associated metadata. Numerical descriptors are generated for features found in a multimedia clip, signatures are generated from these descriptors, and a reference signature database is constructed from these signatures. Query signatures are also generated for a query multimedia clip. These query signatures are searched against the reference database using a fast similarity search procedure, to produce a candidate list of matching signatures. This candidate list is further analyzed to find the most likely reference matches. Signature correlation is performed between the likely reference matches and the query clip to improve detection accuracy.
1. A computing system configured for performing a set of acts comprising:
generating an N-dimension reference signature and a traversal hash reference signature based on an N-dimension reference descriptor for a reference media content feature identified in a frame of a reference video sequence;
storing the N-dimension reference signature and a link to the reference media content feature at a location in a reference database that is addressable by the traversal hash reference signature;
obtaining an N-dimension query signature and a traversal hash query signature, wherein the N-dimension query signature and the traversal hash query signature are based on an N-dimension query descriptor for a query media content feature of a video sequence; and
searching the reference database using the traversal hash query signature to select reference signatures that are within a specified signature distance from the N-dimension query signature.
2. The computing system of claim 1 , wherein the selected reference signatures are marked as candidate reference signatures and the reference media content feature is a reference image feature.
3. The computing system of claim 2 , wherein the set of acts further comprises:
sorting the candidate reference signatures according to a count of times that a closely matching signature appears in the candidate reference signatures to find a likely matching reference media content feature, wherein the reference media content feature is a member of a plurality of reference media content features stored in the reference database.
4. The computing system of claim 2 , wherein the set of acts further comprises:
determining a plurality of scores, wherein each score is based on a distance between the N-dimension query signature and a different one of the candidate reference signatures; and
comparing the plurality of scores with a score threshold to determine whether a reference media content feature associated with a score is a true positive match or a false positive match.
5. The computing system of claim 2 , wherein the set of acts further comprises:
counting a frequency of occurrence of multimedia clip identifications associated with each closely matching signature in the candidate reference signatures; and
sorting by the frequency of occurrence to find T likely reference multimedia clip matches, wherein T is at least one.
6. The computing system of claim 1 , wherein generating the traversal hash reference signature comprises:
combining multiple dimensions of the N-dimension reference descriptor to form combined values; and
comparing the combined values with a threshold to form the traversal hash reference signature.
7. The computing system of claim 1 , wherein generating the N-dimension reference signature comprises quantizing the N-dimension reference descriptor with a threshold to form the N-dimension reference signature.
8. A computing system configured for performing a set of acts comprising:
generating on a portable device an N-dimension reference signature and a traversal hash signature based on an N-dimension reference descriptor for a reference image feature identified in a frame of a reference video sequence;
storing the N-dimension reference signature and a link to the reference image feature at a location in reference database on the portable device that is addressable by the traversal hash signature;
generating on the portable device an N-dimension query signature and a traversal hash query signature based on an N-dimension query descriptor for a query image feature of a video sequence; and
searching the reference database on the portable device using the traversal hash query signature to select reference signatures that are within a specified signature distance from the N-dimension query signature, wherein the selected reference signatures are marked as candidate reference signatures.
9. The computing system of claim 8 , wherein the set of acts further comprises:
sorting the candidate reference signatures according to a count of times that a closely matching signature appears in the candidate reference signatures to find a likely matching reference media content feature, wherein the reference media content feature is a member of a plurality of reference media content features stored in the reference database.
10. The computing system of claim 8 , wherein the set of acts further comprises:
determining a plurality of scores, wherein each score is based on a distance between the N-dimension query signature and a different one of the candidate reference signatures; and
comparing the plurality of scores with a score threshold to determine whether a reference media content feature associated with a score is a true positive match or a false positive match.
11. The computing system of claim 8 , wherein the set of acts further comprises:
counting a frequency of occurrence of multimedia clip identifications associated with each closely matching signature in the candidate reference signatures; and
sorting by the frequency of occurrence to find T likely reference multimedia clip matches, wherein T is at least one.
12. The computing system of claim 8 , wherein generating the traversal hash reference signature comprises:
combining multiple dimensions of the N-dimension reference descriptor to form combined values; and
comparing the combined values with a threshold to form the traversal hash reference signature.
13. The computing system of claim 8 , wherein generating the N-dimension reference signature comprises quantizing the N-dimension reference descriptor with a threshold to form the N-dimension reference signature.
14. The computing system of claim 8 , wherein the set of acts further comprises determining the N-dimension query descriptor by partitioning the query image into sub-regions and determining pixel features within each sub-region.
15. A computing system configured for performing a set of acts comprising:
storing a plurality of N-dimension reference signatures in a reference signature database, wherein each reference signature is stored with a link to a corresponding reference image;
generating an N-dimension query signature and a traversal hash query signature for a query image; and
searching the reference signature database using the traversal hash query signature as an address into the reference signature database to find similar signatures that are within a specified signature distance R from each other, wherein the similar signatures represent reference images that are similar to the query image.
16. The computing system of claim 15 , wherein the set of acts further comprises:
quantizing dimensions of an N-dimension reference descriptor to form an N-dimension reference signature; and
quantizing dimensions of the N-dimension reference signature to form a traversal hash reference signature, wherein the traversal hash reference signature points to a starting location in the reference signature database at which a cluster of related reference signatures are to be stored.
17. The computing system of claim 15 , wherein the traversal hash query signature is generated based on a quantization of dimensions of an N-dimension query descriptor used to form the N-dimension query signature.
18. The computing system of claim 17 , wherein the set of acts further comprises determining the N-dimension query descriptor by partitioning the query image into sub-regions and determining pixel features within each sub-region.
19. The computing system of claim 18 , wherein the pixel features within each sub-region comprise x-gradient values based on pixel intensity differences in an x-direction and y-gradient values based on pixel intensity differences in a y-direction.
20. The computing system of claim 17 , wherein the sub-regions comprise a Bx by By grid of sub-regions.