Media fingerprinting and identification system
The overall architecture and details of a scalable video fingerprinting and identification system that is robust with respect to many classes of video distortions is described. In this system, a fingerprint for a piece of multimedia content is composed of a number of compact signatures, along with traversal hash signatures and associated metadata. Numerical descriptors are generated for features found in a multimedia clip, signatures are generated from these descriptors, and a reference signature database is constructed from these signatures. Query signatures are also generated for a query multimedia clip. These query signatures are searched against the reference database using a fast similarity search procedure, to produce a candidate list of matching signatures. This candidate list is further analyzed to find the most likely reference matches. Signature correlation is performed between the likely reference matches and the query clip to improve detection accuracy.
1. A method comprising:
obtaining, by one or more computing devices, a pair of query signatures corresponding to a query sequence of media content;
searching, by the one or more computing devices, a reference database, using the pair of query signatures so as to identify a set of reference signatures that are within a distance measure of the pair of query signatures;
determining, by the one or more computing devices, a correlation score between the pair of query signatures and a pair of reference signatures of the set of reference signatures, wherein the correlation score is based on a first similarity score between a first query signature of the pair of query signatures and a first reference signature of the pair of reference signatures, a second similarity score between a second query signature of the pair of query signatures and a second reference signature of the pair of reference signatures, and a frame correlation between the pair of query signatures and the pair of reference signatures; and
based at least on the correlation score, identifying, by the one or more computing devices, a reference media content corresponding to the query sequence of media content.
2. The method of claim 1 , wherein the first query signature and the second query signature corresponding to different frames of the query sequence of media content.
3. The method of claim 1 , wherein searching the reference database comprises searching the reference database using a hash index of the first query signature and a hash index of the second query signature.
4. The method of claim 1 , wherein identifying the reference media content comprises identifying the reference media content based at least on the correlation score satisfying a threshold.
5. The method of claim 1 , further comprising determining a correlation in space between the pair of query signatures and the pair of reference signatures,
wherein the identifying is further based on the correlation in space.
6. The method of claim 5 , wherein the correlation in space is based on spatial locations of the first query signature, the second query signature, the first reference signature, and the second reference signature.
7. The method of claim 1 , wherein identifying the reference media content comprises identifying a name of the reference media content and a start frame within the reference media content.
8. A non-transitory computer-readable medium having stored therein instructions that are executable to cause a computing system to perform functions comprising:
obtaining a pair of query signatures corresponding to a query sequence of media content,
searching a reference database using the pair of query signatures so as to identify a set of reference signatures that are within a distance measure of the pair of query signatures;
determining a correlation score between the pair of query signatures and a pair of reference signatures of the set of reference signatures, wherein the correlation score is based on a first similarity score between a first query signature of the pair of query signatures and a first reference signature of the pair of reference signatures, a second similarity score between a second query signature of the pair of query signatures and a second reference signature of the pair of reference signatures, and a frame correlation between the pair of query signatures and the pair of reference signatures; and
based at least on the correlation score, identifying a reference media content corresponding to the query sequence of media content.
9. The non-transitory computer-readable medium of claim 8 , wherein the first query signature and the second query signature corresponding to different frames of the query sequence of media content.
10. The non-transitory computer-readable medium of claim 8 , wherein searching the reference database comprises searching the reference database using a hash index of the first query signature and a hash index of the second query signature.
11. The non-transitory computer-readable medium of claim 8 , wherein identifying the reference media content comprises identifying the reference media content based at least on the correlation score satisfying a threshold.
12. The non-transitory computer-readable medium of claim 8 :
wherein the functions further comprise determining a correlation in space between the pair of query signatures and the pair of reference signatures, and
wherein the identifying is further based on the correlation in space.
13. The non-transitory computer-readable medium of claim 12 , wherein the correlation in space is based on spatial locations of the first query signature, the second query signature, the first reference signature, and the second reference signature.
14. The non-transitory computer-readable medium of claim 8 , wherein identifying the reference media content comprises identifying a name of the reference media content and a start frame within the reference media content.
15. A computing system comprising:
a processing unit; and
a computer-readable medium having stored therein instructions that are executable to cause the computing system to perform functions comprising:
obtaining a pair of query signatures corresponding to a query sequence of media content,
searching a reference database using the pair of query signatures so as to identify a set of reference signatures that are within a distance measure of the pair of query signatures;
determining a correlation score between the pair of query signatures and a pair of reference signatures of the set of reference signatures, wherein the correlation score is based on a first similarity score between a first query signature of the pair of query signatures and a first reference signature of the pair of reference signatures, a second similarity score between a second query signature of the pair of query signatures and a second reference signature of the pair of reference signatures, and a frame correlation between the pair of query signatures and the pair of reference signatures; and
based at least on the correlation score, identifying a reference media content corresponding to the query sequence of media content.
16. The computing system of claim 15 , wherein the first query signature and the second query signature corresponding to different frames of the query sequence of media content.
17. The computing system of claim 15 , wherein searching the reference database comprises searching the reference database using a hash index of the first query signature and a hash index of the second query signature.
18. The computing system of claim 15 , wherein identifying the reference media content comprises identifying the reference media content based at least on the correlation score satisfying a threshold.
19. The computing system of claim 15 :
wherein the functions further comprise determining a correlation in space between the pair of query signatures and the pair of reference signatures, and
wherein the identifying is further based on the correlation in space.
20. The computing system of claim 15 , wherein identifying the reference media content comprises identifying a name of the reference media content and a start frame within the reference media content.