Scalable, adaptable, and manageable system for multimedia identification
An architecture for a multimedia search system is described. To perform similarity matching of multimedia query frames against reference content, reference database comprising of a cluster index using cluster keys to perform similarity matching and a multimedia index to perform sequence matching is built. Methods to update and maintain the reference database that enables addition and removal of the multimedia contents, including portions of multimedia content, from the reference database in a running system are described. Hierarchical multi-level partitioning methods to organize the reference database are presented. Smart partitioning of the reference multimedia content according to the nature of the multimedia content, and according to the popularity among the social media, that supports scalable fast multimedia identification is also presented. A caching mechanism for multimedia search queries in a centralized or in a decentralized distributed system and a client based local multimedia search system enabling multimedia tracking are described.
1. A method of tracking units of multimedia content, the method comprising:
sending a search query from a client device to a search server that is located remote from the client device, wherein the search query is used to search in a remote database coupled to the search server for units of multimedia content of a multimedia program that is currently displayed for viewing on the client device;
storing reference signatures, retrieved from the remote database, in a local database on the client device, in response to a match on the search query, the reference signatures including signatures for part of the multimedia program around the searched for units of multimedia content and including matching signatures for the searched for units of multimedia content;
detecting matches in query results in the local database for succeeding units of multimedia content using a track search approach to search each succeeding unit of a pre-specified duration of multimedia content of the multimedia program for a match in the local database;
blocking display of the succeeding units of multimedia content on the client device representing a blocked display upon a no match indication detected in the query results; and
displaying subsequent units of multimedia content on the client device after the blocked display upon a match indication detected in the query results by the track search approach.
2. The method of claim 1 , wherein the track search approach uses a correlation approach to determine a match.
3. The method of claim 1 , further comprising:
searching the reference signatures stored in the client device using a full search query approach upon a no match indication detected in the query results.
4. The method of claim 3 , wherein the full search query approach uses a similarity search and a correlation search approach to determine a match.
5. The method of claim 1 , wherein each reference signature includes a cluster key comprising a link reference address (LRA) to a list of signature records stored in the local database.
6. The method of claim 5 , wherein the cluster key is generated by applying a fingerprinting process to bits of a reference fingerprint having a first number of bits, wherein the cluster key comprises a second number of bits that is less than the first number of bits.
7. The method of claim 1 further comprising:
sending a second search query to the search server for the subsequent units of multimedia content of the multimedia program that is currently displayed for viewing on the client device; and
sending reference signatures for the full multimedia program from the remote database to the client device to be stored in the local database.
8. An apparatus for tracking units of multimedia content, the apparatus comprising:
a client device configured for sending a search query from the client device to a search server that is located remote from the client device, wherein the search query is used to search in a remote database coupled to the search server for units of multimedia content of a multimedia program that is currently displayed for viewing on the client device; and
a local database on the client device configured for storing reference signatures, retrieved from the remote database, in the local database on the client device, in response to a match on the search query, the reference signatures including signatures for part of the multimedia program around the searched for units of multimedia content and including matching signatures for the searched for units of multimedia content,
wherein matches are detected in query results in the local database for succeeding units of multimedia content using a track search approach to search each succeeding unit of a pre-specified duration of multimedia content of the multimedia program for a match in the local database,
wherein display of the succeeding units of multimedia content are blocked on the client device representing a blocked display upon a no match indication detected in the query results, and
wherein subsequent units of multimedia content are displayed on the client device after the blocked display upon a match indication detected in the query results by the track search approach.
9. The apparatus of claim 8 , wherein the reference signatures are searched in the client device using a full search query approach upon a no match indication detected in the query results.
10. The apparatus of claim 9 , wherein the full search query approach uses a similarity search and a correlation search approach to determine a match.
11. The apparatus of claim 8 , wherein each reference signature includes a cluster key comprising a link reference address (LRA) to a list of signature records stored in the local database.
12. The apparatus of claim 11 , wherein the cluster key is generated by applying a fingerprinting process to bits of a reference fingerprint having a first number of bits, wherein the cluster key comprises a second number of bits that is less than the first number of bits.
13. The apparatus of claim 8 , wherein a second search query is sent to the search server for the subsequent units of multimedia content of the multimedia program that is currently displayed for viewing on the client device and wherein the reference signatures for the full multimedia program are sent from the remote database to the client device to be stored in the local database.
14. A computer readable non-transitory medium storing a computer program which causes a computer system to perform a method of tracking units of multimedia content, the method comprising:
sending a search query from a client device to a search server that is located remote from the client device, wherein the search query is used to search in a remote database coupled to the search server for units of multimedia content of a multimedia program that is currently displayed for viewing on the client device;
storing reference signatures, retrieved from the remote database, in a local database on the client device, in response to a match on the search query, the reference signatures including signatures for part of the multimedia program around the searched for units of multimedia content and including matching signatures for the searched for units of multimedia content;
detecting matches in query results in the local database for succeeding units of multimedia content using a track search approach to search each succeeding unit of a pre-specified duration of multimedia content of the multimedia program for a match in the local database;
blocking display of the succeeding displayed units of multimedia content on the client device representing a blocked display upon a no match indication detected in the query results; and
displaying subsequent units of multimedia content on the client device after the blocked display upon a match indication detected in the query results by the track search approach.
15. The computer readable non-transitory medium method of claim 14 , wherein the track search approach uses a correlation approach to determine a match.
16. The computer readable non-transitory medium method of claim 14 further comprising:
searching the reference signatures stored in the client device using a full search query approach upon a no match indication detected in the query results.
17. The computer readable non-transitory medium method of claim 16 , wherein the full search query approach uses a similarity search and a correlation search approach to determine a match.
18. The computer readable non-transitory medium method of claim 14 , wherein each reference signature includes a cluster key comprising a link reference address (LRA) to a list of signature records stored in the local database.
19. The computer readable non-transitory medium method of claim 18 , wherein the cluster key is generated by applying a fingerprinting process to bits of a reference fingerprint having a first number of bits, wherein the cluster key comprises a second number of bits that is less than the first number of bits.
20. The computer readable non-transitory medium method of claim 14 further comprising:
sending a second search query to the search server for the subsequent units of multimedia content of the multimedia program that is currently displayed for viewing on the client device; and
sending reference signatures for the full multimedia program from the remote database to the client device to be stored in the local database.