Media fingerprinting and identification system
The overall architecture and details of a scalable video fingerprinting and identification system that is robust with respect to many classes of video distortions is described. In this system, a fingerprint for a piece of multimedia content is composed of a number of compact signatures, along with traversal hash signatures and associated metadata. Numerical descriptors are generated for features found in a multimedia clip, signatures are generated from these descriptors, and a reference signature database is constructed from these signatures. Query signatures are also generated for a query multimedia clip. These query signatures are searched against the reference database using a fast similarity search procedure, to produce a candidate list of matching signatures. This candidate list is further analyzed to find the most likely reference matches. Signature correlation is performed between the likely reference matches and the query clip to improve detection accuracy.
1. A method of generating signatures from features in a sequence of frames of a media clip executed on one or more hardware devices, the method comprising:
generating a plurality of N-dimensional signatures for features detected in a frame and in a following frame of the media clip;
computing signature bit distances between respective N-dimensional signatures in the frame and in the following frame of the media clip for a robustness analysis;
eliminating N-dimensional signatures from the plurality of N-dimensional signatures that have an associated signature bit distance greater than a first robustness threshold R that are not temporally robust to create a set of remaining N-dimensional signatures that are temporally robust in representing the media clip; and
storing N-dimensional signatures of the set of remaining N-dimensional signatures and an identifier of the media clip at locations in a reference signature database.
2. The method of claim 1 further comprising:
generating N-dimensional descriptors based on a matrix of pixels for each detected feature of a plurality of features detected in the frame and in the following frame of the media clip; and
quantizing the N-dimensional descriptors with a first threshold to create the N-dimensional signatures for each feature detected in the frame and in the following frame of the media clip.
3. The method of claim 2 further comprising:
computing N digit values comprising the N-dimensional descriptors from pixel intensity values for each detected feature based on the matrix of pixels placed at center coordinates (x, y) of each detected feature, wherein the matrix of pixels is configured with a spatial extent determined from an x direction scale factor (sx) and a y direction scale factor (sy).
4. The method of claim 1 further comprising:
sorting the remaining N-dimensional signatures having signature bit distances that are less than or equal to R according to the signature bit distances; and
selecting a number T of remaining N-dimensional signatures from the sorted remaining N-dimensional signatures as having a sufficiently high measure of strength in representing the media clip.
5. The method of claim 1 further comprising:
generating N-dimensional descriptors based on selected image pixels of features detected in the frame and in the following frame of the media clip;
combining dimensions of the N-dimensional descriptors to create combined dimensional descriptors; and
quantizing the combined dimensional descriptors with a second threshold to create traversal hash signatures for the features detected in the frame and in the following frame of the media clip.
6. The method of claim 1 further comprising:
generating traversal hash signatures, that are representations of each N-dimensional signatures, for the features detected in the frame and in the following frame of the media clip;
computing a hash bit distance between respective traversal hash signatures in the frame and in the following frame of the media clip for a second robustness analysis; and
eliminating N-dimensional signatures from the set of remaining N-dimensional signatures that have an associated hash bit distance greater than a second robustness threshold Q that are not temporally robust.
7. The method of claim 1 further comprising:
computing an average distance between a detected feature center location and center locations for M near detected features to determine a spatial diversity value, wherein the average distance indicates how close detected features are to each other within the frame and within the following frame and wherein the M near detected features satisfy a nearness determination; and
eliminating N-dimensional signatures from the set of remaining N-dimensional signatures that are too close based on an established threshold for closeness.
8. A method of generating hash signatures from features in a sequence of frames of a media clip executed on one or more hardware devices, the method comprising:
generating a plurality of N-dimensional descriptors based on selected image pixels of a plurality of features detected in a frame and in a following frame of the media clip;
quantizing combined dimensions of the N-dimensional descriptors with a first threshold to create a plurality of hash signatures for the features detected in the frame and in the following frame of the media clip;
computing hash bit distances between respective hash signatures in the frame and in the following frame of the media clip for a robustness analysis;
eliminating hash signatures from the plurality of hash signatures that have an associated hash bit distance greater than a robustness threshold Q that are not temporally robust to create a set of remaining hash signatures that are temporally robust for accessing the media clip; and
storing hash signatures of the set of remaining hash signatures and an identifier of the media clip at locations in a reference signature database.
9. The method of claim 8 further comprising:
generating the N-dimensional descriptors based on a matrix of pixels for each detected feature of the plurality of features detected in the frame and in the following frame of the media clip; and
quantizing the N-dimensional descriptors with a first threshold to create N-dimensional signatures for each feature detected in the frame and in the following frame of the media clip.
10. The method of claim 9 further comprising:
storing hash bit distances that are less than or equal to Q with an associated N-dimensional signature in a media database;
sorting the N-dimensional signatures according to the hash bit distances; and
selecting a number T of N-dimensional signatures from the sorted N-dimensional signatures as having a sufficiently high measure of strength in representing the media clip.
11. The method of claim 8 further comprising:
computing N digit values comprising the N-dimensional descriptors from pixel intensity values for each detected feature based on a matrix of pixels placed at center coordinates (x, y) of each detected feature, wherein the matrix of pixels is configured with a spatial extent determined from an x direction scale factor (sx) and a y direction scale factor (sy).
12. The method of claim 8 further comprising:
computing an average distance between a detected feature center location and center locations for M near detected features to determine a spatial diversity value, wherein the average distance indicates how close detected features are to each other within the frame and within the following frame and wherein the M near detected features satisfy a nearness determination; and
eliminating hash signatures from the set of remaining hash signatures that are too close based on an established threshold for closeness.
13. A method of generating signatures from features in a sequence of frames of a media clip executed on one or more hardware device, the method comprising:
generating a first plurality of N-dimensional signatures for features detected in a frame of the media clip;
generating a second plurality of N-dimensional signatures for features detected in a following frame of the media clip;
computing signature bit distances between respective N-dimensional signatures in the first plurality of N-dimensional signatures and in the second plurality of N-dimensional signatures for a robustness analysis;
eliminating N-dimensional signatures from the first plurality of N-dimensional signatures that have an associated signature bit distance greater than a first robustness threshold R that are not temporally robust to create a set of remaining N-dimensional signatures that are temporally robust in representing the media clip; and
storing N-dimensional signatures from the set of remaining N-dimensional signatures and an identifier of the media clip at locations in a reference signature database.
14. The method of claim 13 , wherein the first plurality of N-dimensional signatures generated for features detected in the frame of the media clip are reduced by the elimination of N-dimensional signatures that are not temporally robust as determined by the robustness analysis between the frame and the following frame.
15. A method to search a reference media database using query signatures generated from features in a sequence of frames of a query media clip executed on one or more hardware devices, the method comprising:
generating a plurality of query signatures and their associated traversal hash signatures for features detected in a frame and in a following frame of the query media clip;
computing query signature bit distances between respective query signatures in the frame and in the following frame of the query media clip for a robustness analysis;
eliminating query signatures and their associated traversal hash signatures from the plurality of query signatures that have an associated query signature bit distance greater than a first robustness threshold R to create a set of remaining query signatures and their associated remaining traversal hash signatures that have a measure of robustness that satisfies the first robustness threshold R as a criteria for representing the query media clip;
storing in a media database a number T of the remaining query signatures having the highest measure of robustness of the remaining query signatures and their associated traversal hash signatures; and
fetching reference signature records using the T remaining traversal hash signatures as addresses from the reference media database to search for candidate matching reference media clips.
16. The method of claim 15 further comprising:
storing the query signature bit distances that are less than or equal to R with the associated remaining query signatures in the media database; and
selecting the T remaining query signatures from the sorted remaining query signatures as having a high measure of strength according to the stored query signature bit distances in representing the query media clip.
17. The method of claim 15 further comprising:
computing an average distance between a detected feature center location and center locations for M near detected features to determine a spatial diversity value, wherein the average distance indicates how close detected features are to each other within the frame and within the following frame and wherein the M near detected features satisfy a nearness determination; and
eliminating query signatures from the set of remaining query signatures that are too close based on an established threshold for closeness.
18. The method of claim 15 further comprising:
storing multimedia clip identifications (IDs) accessed from the reference media database with the candidate matching reference media clips in the media database; and
counting the number of times a multimedia clip ID appears in the candidate matching reference media clips to determine most likely top matching reference media clips.
19. The method of claim 15 , wherein the query media clip and the reference media clip comprise pixel data and audio waveform data.
20. The method of claim 15 , wherein the media database and the reference media database are located on a portable user device.