IP Library Granted Patent US 11,487,814
Granted Patent B2
US 11,487,814 · App. 17/140,992 · Granted Nov 1, 2022

Multiple stage indexing of audio content

Inventors: Peter C. DiMaria (Berkeley, CA); Markus K. Cremer (Orinda, CA); Barnabas Mink (San Francisco, CA); Tanji Koshio (Emeryville, CA); Kei Tsuji (Emeryville, CA)
Assignee: Gracenote, Inc.
G06F16/683G06F16/61
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,487,814
App. No.
17/140,992
Granted
Nov 1, 2022
Kind
B2
Abstract

Techniques of content unification are disclosed. In some example embodiments, a computer-implemented method comprises: determining clusters based a comparison of a plurality of audio content using a first matching criteria, each cluster of the plurality of clusters comprising at least two audio content from the plurality of audio content; for each cluster of the plurality of clusters, determining a representative audio content for the cluster from the at least two audio content of the cluster; loading the corresponding representative audio content of each cluster into an index; matching the query audio content to one of the representative audio contents using a first matching criteria; determining the corresponding cluster of the matched representative audio content; and identifying a match between the query audio content and at least one of the audio content of the cluster of the matched representative audio content based on a comparison using a second matching criteria.

Claims (43)

1. A computer-implemented method comprising:

determining, by at least one hardware processor, a plurality of clusters based on a comparison of a plurality of audio contents using a first matching criteria, each cluster of the plurality of clusters comprising at least two audio contents from the plurality of audio contents;

for each cluster of the plurality of clusters, determining, by the at least one hardware processor, a representative audio content for the cluster from the at least two audio contents of the cluster;

loading, by the at least one hardware processor, the corresponding representative audio content of each cluster in the plurality of clusters into an index, the index including the representative audio content stored in association with a hash value, wherein the hash value is associated with a candidate reference identifier;

removing, by the at least one hardware processor, candidate reference identifiers that appear less than a threshold number of times, a remaining set of candidate reference identifiers retained for comparison;

generating a first comparison, by the at least one hardware processor, of a query audio content to each one of the representative audio contents associated with the remaining set of candidate reference identifiers, the first comparison generated using the first matching criteria, the first comparison including a comparison of at least one of a content duration ratio, a bit error rate at a matching location, or a length of matching positions; and

matching, by the at least one hardware processor, the query audio content to one of the representative audio contents based on the generated first comparison;

wherein the determining the plurality of clusters comprises comparing fingerprint data of each of the plurality of audio contents using the first matching criteria, the comparison including a granular comparison based on a sub-cluster of each of the plurality of clusters.

2. The computer-implemented method of claim 1 , wherein the hash value is based on permutations of a binary vector formed using a spectral representation of the audio content.

3. The computer-implemented method of claim 1 , further including determining, by the at least one hardware processor, the corresponding cluster of the matched one of the representative audio contents.

4. The computer-implemented method of claim 3 , further including generating a second comparison, by the at least one hardware processor, of the query audio content to each one of the at least two audio contents of the determined corresponding cluster of the matched one of the representative audio contents using a second matching criteria different from the first matching criteria.

5. The computer-implemented method of claim 4 , further including identifying, by the at least one hardware processor, a match between the query audio content and at least one of the audio contents of the determined corresponding cluster of the matched one of the representative audio contents based on the generated second comparison of the determined corresponding cluster using the first and the second matching criteria, the match used to determine similarity of the queried audio content to the representative audio content.

6. The computer-implemented method of claim 5 , wherein comparing the query audio content to each one of the at least two audio contents of the determined corresponding cluster comprises comparing fingerprint data of the query audio content with fingerprint data of each one of the at least two audio contents of the determined corresponding cluster using the second matching criteria.

7. The computer-implemented method of claim 6 , wherein the fingerprint data comprises a micro-fingerprint or a full fingerprint.

8. The computer-implemented method of claim 1 , wherein the fingerprint data comprises a nano-fingerprint.

9. The computer-implemented method of claim 1 , wherein the corresponding representative audio content of each cluster in the plurality of clusters is the only audio content of the cluster that is loaded into the index.

10. The computer-implemented method of claim 1 , wherein the matching of the query audio content to one of the representative audio contents comprises comparing fingerprint data of the query audio content with fingerprint data of each of the representative audio contents in the index using the first matching criteria.

11. The computer-implemented method of claim 1 , wherein each one of the plurality of audio contents comprises a song.

12. A system comprising:

at least one processor; and

a non-transitory computer-readable medium storing executable instructions that, when executed, cause the at least one processor to perform operations comprising:

determining a plurality of clusters based on a comparison of a plurality of audio contents using a first matching criteria, each cluster of the plurality of clusters comprising at least two audio contents from the plurality of audio contents;

for each cluster of the plurality of clusters, determining a representative audio content for the cluster from the at least two audio contents of the cluster;

loading the corresponding representative audio content of each cluster in the plurality of clusters into an index, the index including the representative audio content stored in association with a hash value, wherein the hash value is associated with a candidate reference identifier;

removing candidate reference identifiers that appear less than a threshold number of times, a remaining set of candidate reference identifiers retained for comparison;

generating a first comparison of a query audio content to each one of the representative audio contents associated with the remaining set of candidate reference identifiers, the first comparison generated using the first matching criteria, the first comparison including a comparison of at least one of a content duration ratio, a bit error rate at a matching location, or a length of matching positions; and

matching the query audio content to one of the representative audio contents based on the generated first comparison;

wherein the determining the plurality of clusters comprises comparing fingerprint data of each of the plurality of audio contents using the first matching criteria, the comparison including a granular comparison based on a sub-cluster of each of the plurality of clusters.

13. The system of claim 12 , wherein the hash value is based on permutations of a binary vector formed using a spectral representation of the audio content.

14. The system of claim 12 , further including determining the corresponding cluster of the matched one of the representative audio contents.

15. The system of claim 14 , further including generating a second comparison of the query audio content to each one of the at least two audio contents of the determined corresponding cluster of the matched one of the representative audio contents using a second matching criteria different from the first matching criteria.

16. The system of claim 15 , further including identifying a match between the query audio content and at least one of the audio contents of the determined corresponding cluster of the matched one of the representative audio contents based on the generated second comparison of the determined corresponding cluster using the first and the second matching criteria, the match used to determine similarity of the queried audio content to the representative audio content.

17. The system of claim 16 , wherein comparing the query audio content to each one of the at least two audio contents of the determined corresponding cluster comprises comparing fingerprint data of the query audio content with fingerprint data of each one of the at least two audio contents of the determined corresponding cluster using the second matching criteria.

18. A non-transitory machine-readable storage medium, tangibly embodying a set of instructions that, when executed by at least one processor, causes the at least one processor to perform operations comprising:

determining a plurality of clusters based on a comparison of a plurality of audio contents using a first matching criteria, each cluster of the plurality of clusters comprising at least two audio contents from the plurality of audio contents;

for each cluster of the plurality of clusters, determining a representative audio content for the cluster from the at least two audio contents of the cluster;

loading the corresponding representative audio content of each cluster in the plurality of clusters into an index, the index including the representative audio content stored in association with a hash value, wherein the hash value is associated with a candidate reference identifier;

removing candidate reference identifiers that appear less than a threshold number of times, a remaining set of candidate reference identifiers retained for comparison;

generating a first comparison of a query audio content to each one of the representative audio contents associated with the remaining set of candidate reference identifiers, the first comparison generated using the first matching criteria, the first comparison including a comparison of at least one of a content duration ratio, a bit error rate at a matching location, or a length of matching positions; and

matching the query audio content to one of the representative audio contents based on the generated first comparison;

wherein the determining the plurality of clusters comprises comparing fingerprint data of each of the plurality of audio contents using the first matching criteria, the comparison including a granular comparison based on a sub-cluster of each of the plurality of clusters.

19. The storage medium according to claim 18 , wherein the instructions cause the processor to determine the corresponding cluster of the matched one of the representative audio contents.

20. The storage medium according to claim 18 , wherein the instructions cause the processor to generate a second comparison of the query audio content to each one of the at least two audio contents of the determined corresponding cluster of the matched one of the representative audio contents using a second matching criteria different from the first matching criteria.

Assignments (4)
SECURITY INTEREST Recorded May 8, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: ARES CAPITAL CORPORATION
Reel/Frame 063574/0632 →
SECURITY INTEREST Recorded Apr 28, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: CITIBANK, N.A.
Reel/Frame 063561/0381 →
SECURITY AGREEMENT Recorded Jan 31, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: BANK OF AMERICA, N.A.
Reel/Frame 063560/0547 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 20, 2021
From: DIMARIA, PETER C.; CREMER, MARKUS K.; MINK, BARNABAS; KOSHIO, TANJI; TSUJI, KEI
To: GRACENOTE, INC.
Reel/Frame 054972/0230 →
Continuity (2)
Continuation 15475459 · Mar 31, 2017
Related Publication 20210124776A1 · Apr 29, 2021