Media fingerprinting and identification system
The overall architecture and details of a scalable video fingerprinting and identification system that is robust with respect to many classes of video distortions is described. In this system, a fingerprint for a piece of multimedia content is composed of a number of compact signatures, along with traversal hash signatures and associated metadata. Numerical descriptors are generated for features found in a multimedia clip, signatures are generated from these descriptors, and a reference signature database is constructed from these signatures. Query signatures are also generated for a query multimedia clip. These query signatures are searched against the reference database using a fast similarity search procedure, to produce a candidate list of matching signatures. This candidate list is further analyzed to find the most likely reference matches. Signature correlation is performed between the likely reference matches and the query clip to improve detection accuracy.
1. A method for creating a media database, the method comprising:
generating an N-dimension descriptor for a selected image feature in a selected frame of a video clip, the N-dimension descriptor being determined from a matrix of pixels placed on the selected image feature at {center x, center y} coordinates and having a spatial extent determined from an x direction scale factor sx and a y direction scale factor sy, wherein the N-dimension descriptor comprises N digit values arranged in H groups of D dimensions;
applying a first threshold to the N-dimension descriptor to generate an N-dimension signature having N binary values;
generating a combined descriptor based on the N-dimension descriptor by combining the digit values in each group of D dimensions to create a combined digit value for each group, wherein the combined descriptor comprises H=N/D combined digit values; and
applying a second threshold to each combined digit value of the combined descriptor to generate a traversal hash signature having H binary values, wherein the traversal hash signature, associated N-dimension signature, associated N-dimension descriptor, and associated {center x, center y, sx, sy} values are stored in a media database.
2. The method of claim 1 , wherein the matrix of pixels is centered on a center x coordinate and a center y coordinate of the selected image feature and with a spatial extent of the matrix of pixels having a size of A*sx by B*sy, wherein A and B are predetermined factors.
3. The method of claim 1 further comprising:
resampling the matrix of pixels configured as a Bx by By matrix having Bx*By blocks, each block configured with an Rx by Ry sub-region of pixel values, to generate a grid with (Bx*By)*(Rx*Ry) pixel values that is used to generate the N-dimension descriptor.
4. The method of claim 3 further comprising:
generating a set of values for each Rx by Ry sub-region of pixel values based on a difference from each pixel in the Rx by Ry sub-region to several other pixels, wherein the set of values for each block of the Bx*By blocks concatenated together form the N-dimension descriptor.
5. The method of claim 1 further comprising:
generating a second N-dimension descriptor for the selected image feature, the second N-dimension descriptor is determined from a second matrix of pixels placed on the selected image feature at {center x+dx, center y+dy} coordinates and having a spatial extent determined from the sx and the sy scale factors, wherein the second N-dimension descriptor comprises N digit values arranged in H groups of D dimensions;
applying the first threshold to the second N-dimension descriptor to generate a second N-dimension signature having N binary values;
generating a second combined descriptor based on the second N-dimension descriptor by combining the digit values in each group of D dimensions of the second N-dimension descriptor to create a combined digit value for each group, wherein each second combined descriptor comprises H=N/D combined digit values; and
applying the second threshold to each combined digit value of the second combined descriptor to generate a second traversal hash signature having H binary values, wherein the second traversal hash signature, associated second N-dimension signature, associated second N-dimension descriptor, and associated {center x+dx, center y+dy, sx, sy}values are stored in a media database.
6. The method of claim 5 , wherein a third matrix of pixels is placed on the selected image feature at {center x−dx, center y−dy} coordinates and having a spatial extent determined from the sx and the sy scale factors to determine a third N-dimension descriptor, a third N-dimension signature, and a third traversal has signature to be stored in the media database.
7. The method of claim 5 , wherein dx and dy are determined from an expected error margin due to the effects of image distortions.
8. The method of claim 1 further comprising:
normalizing each digit value of each N-dimension descriptor to form N normalized digit values representing a normalized N-dimension descriptor; and
generating a sum of the N normalized digit values which is stored as descriptor information with the associated N-dimension descriptor in the media database.
9. The method of claim 8 , wherein the descriptor information is determined to be less than a minimum threshold which results in the associated N-dimension descriptor being deleted from the media database.
10. The method of claim 1 further comprising:
generating a count of 1's in each of the N-dimension signatures; and
storing the count of 1's as signature information with the associated N-dimension signature in the media database.
11. The method of claim 10 , wherein the signature information is determined to be less than a minimum threshold which results in the associated N-dimension signature being deleted from the media database.
12. The method of claim 1 further comprising:
generating an average distance measure between the selected image features and a plurality of image features as a measure of closeness that indicates a density of image features according to location; and
storing the measure of closeness with the associated {centerx, centery, sx, sy} values in the media database.
13. The method of claim 12 , wherein the measure of closeness is compared to a closeness threshold to determine signature that are too close based on this threshold are eliminated from the media database.
14. The method of claim 1 further comprising:
generating a temporal robustness measure by calculating first bit distances between the N-dimension signatures of the selected image feature in the selected frame and N-dimension signatures of the selected image feature in a frame that follows the selected frame; and
comparing the first bit distances with a first robustness estimate threshold to produce a first set of temporal robustness values, each value corresponding to each N-dimension binary signature.
15. The method of claim 1 further comprising:
generating a temporal robustness measure by calculating second bit distances between the traversal hash signatures of the selected image feature in the selected frame and traversal hash signatures of the selected image feature in a frame that follows the selected frame; and
comparing the second bit distances with a second robustness estimate threshold to produce a second set of temporal robustness values, each value corresponding to each traversal hash signature.
16. A method for creating a media database, the method comprising:
determining a center x coordinate, a center y coordinate, a scale factor sx in the x direction, and a scale factor sy in the y direction for a selected image feature in a selected frame of a video clip;
generating an N-dimensional descriptor having K digit values for the selected image feature based on a matrix of pixels centered on the center x and center y coordinates and with a spatial extent around the selected image feature determined by dimensions of A*sx by B*sy, wherein A and B are predetermined factors;
applying a first threshold to the N-dimensional descriptor to generate a binary signature having K digit values;
combining bits associated with the N-dimensional descriptor to generate a combined descriptor having H digit values, H<K; and
applying a second threshold to the combined descriptor to generate a reference traversal hash signature having H digit values, wherein the reference traversal hash signature provides a link to the selected image feature and associated media content in a reference media database.
17. The method of claim 16 , wherein the traversal hash signature provides a link to the binary signature, the N-dimensional descriptor, the {center x, center y, sx, sy}, and metadata of the selected image feature in the media database.
18. The method of claim 16 , wherein a search system and the reference media database are combined in a portable device and the portable device identifies a reference traversal hash signature that is a closest match to a query signature providing access to the selected image feature and associated media content in the reference media database.
19. The method of claim 16 , wherein a portable device generates a query signature from a traversal hash signature to search for media content on the reference media database in a server device remote from the portable device.
20. A computer readable non-transitory medium storing a computer program which causes a computer system to perform a method for creating a media database, the method comprising:
generating an N-dimension descriptor for a selected image feature in a selected frame of a video clip, the N-dimension descriptor being determined from a matrix of pixels placed on the selected image feature at {center x, center y} coordinates and having a spatial extent determined from an x direction scale factor sx and a y direction scale factor sy, wherein the N-dimension descriptor comprises N digit values arranged in H groups of D dimensions;
applying a first threshold to the N-dimension descriptor to generate an N-dimension signature having N binary values;
generating a combined descriptor based on the N-dimension descriptor by combining the digit values in each group of D dimensions to create a combined digit value for each group, wherein the combined descriptor comprises H=N/D combined digit values; and
applying a second threshold to each combined digit value of the combined descriptor to generate a traversal hash signature having H binary values, wherein the traversal hash signature, associated N-dimension signature, associated N-dimension descriptor, and associated {center x, center y, sx, sy} values are stored in a media database.