Media fingerprinting and identification system
View Patent ↗The overall architecture and details of a scalable video fingerprinting and identification system that is robust with respect to many classes of video distortions is described. In this system, a fingerprint for a piece of multimedia content is composed of a number of compact signatures, along with traversal hash signatures and associated metadata. Numerical descriptors are generated for features found in a multimedia clip, signatures are generated from these descriptors, and a reference signature database is constructed from these signatures. Query signatures are also generated for a query multimedia clip. These query signatures are searched against the reference database using a fast similarity search procedure, to produce a candidate list of matching signatures. This candidate list is further analyzed to find the most likely reference matches. Signature correlation is performed between the likely reference matches and the query clip to improve detection accuracy.
1. A method for automatically fingerprinting multimedia content received from a distributed source, and search monitoring for information corresponding to the multimedia content on a portable device, the method comprising:
generating on a portable device an N-dimensional query audio descriptor for an audio feature identified in a temporal audio segment of a multimedia clip and an N-dimensional query video descriptor for a video feature identified in a frame of the multimedia clip received on the portable device;
generating on the portable device an N-dimensional query audio signature based on an audio quantization of the N-dimensional query audio descriptor and an N-dimensional query video signature based on a video quantization of the N-dimensional query video descriptor;
generating on the portable device a traversal hash query audio signature based on a combination of dimensions of the N-dimension query audio descriptor and a traversal hash query video signature based on a combination of dimensions of the N-dimensional query video descriptor; and
searching a reference multimedia clip database on the portable device using the traversal hash query audio signature and the traversal hash query video signature to find a set of signatures that are within a specified signature distance to the associated N-dimensional query audio signature and the associated N-dimensional query video signature, wherein the portable device is provided access to data related to multimedia content associated with a likely matching signature selected from the set of signatures.
2. The method of claim 1 further comprising:
generating on a portable device an N-dimensional audio descriptor for each feature identified in a temporal segment of reference audio data received on the portable device;
generating on the portable device an N-dimensional audio signature based on a quantization of the N-dimensional audio descriptor;
generating on the portable device a traversal hash audio signature based on a combination of dimensions of the N-dimension audio descriptor; and
storing each N-dimensional audio signature at a location on the portable device in a reference signature database addressable by the traversal hash audio signature.
3. The method of claim 2 , wherein the traversal hash audio signature includes additional bits representing a time stamp of an underlying audio feature.
4. The method of claim 1 further comprising:
searching a reference multimedia clip database on a remote server accessible by the portable device using the traversal hash audio query signature to find a set of signatures that are within a specified signature distance to the N-dimensional query audio signature, wherein the portable device is provided access to data related to multimedia content associated with a likely matching signature selected from the set of signatures.
5. The method of claim 1 further comprising:
combining multiple dimensions of the N dimension query audio descriptor to form combined values; and
comparing the combined values with a traversal hash audio threshold to form the traversal hash query audio signature.
6. The method of claim 1 further comprising:
sorting the set of signatures according to a count of the number of times a closely matching signature appears in the set of signatures to find likely matching reference multimedia clip signatures.
7. The method of claim 6 further comprising:
determining a plurality of scores, wherein each score is based on a distance between the N dimension query audio signature and the N-dimensional query video signature and a different signatures of the likely matching reference multimedia clip signatures; and
comparing the plurality of scores with a score threshold to determine whether a reference multimedia clip associated with a score is a true positive match or a false positive match.
8. A computer readable non-transitory medium storing a computer program which causes a computer system to perform a method for automatically fingerprinting multimedia content received from a distributed source, and search monitoring for information corresponding to the multimedia content on a portable device, the method comprising:
generating on a portable device an N-dimensional query audio descriptor for an audio feature identified in a temporal audio segment of a multimedia clip and an N-dimensional query video descriptor for a video feature identified in a frame of the multimedia clip received on the portable device;
generating on the portable device an N-dimensional query audio signature based on an audio quantization of the N-dimensional query audio descriptor and an N-dimensional query video signature based on a video quantization of the N-dimensional query video descriptor;
generating on the portable device a traversal hash query audio signature based on a combination of dimensions of the N-dimension query audio descriptor and a traversal hash query video signature based on a combination of dimensions of the N-dimensional query video descriptor; and
searching a reference multimedia clip database on the portable device using the traversal hash query audio signature and the traversal hash query video signature to find a set of signatures that are within a specified signature distance to the associated N-dimensional query audio signature and the associated N-dimensional query video signature, wherein the portable device is provided access to data related to multimedia content associated with a likely matching signature selected from the set of signatures.
9. The computer readable non-transitory medium of claim 8 , wherein the reference multimedia clip database located on the portable device is a subset of a reference multimedia database accessible by the portable device.
10. The computer readable non-transitory medium of claim 8 further comprising:
combining multiple dimensions of the N-dimensional query audio descriptor to form combined values; and
comparing the combined values with a traversal hash audio threshold to form the traversal hash audio signature.
11. The computer readable non-transitory medium of claim 8 further comprising:
combining multiple dimensions of the N-dimensional descriptor to form query combined values; and
comparing the query combined values with a traversal hash audio threshold to form the traversal hash audio signature.
12. The computer readable non-transitory medium of claim 8 , wherein the traversal hash query audio signature includes additional bits representing a time stamp of an underlying audio feature.
13. The computer readable non-transitory medium of claim 8 further comprising:
determining a plurality of audio scores, wherein each audio score is based on a distance between the N-dimensional query audio signature and a different one of likely matching reference multimedia clip audio signatures;
determining a plurality of video scores, wherein each video score is based on a distance between the N-dimensional query video signature and a different one of likely matching reference multimedia clip video signatures; and
comparing the plurality of audio scores and the plurality of video scores with a score threshold to determine whether a reference multimedia clip associated with a score is a true positive match or a false positive match.