IP Library Granted Patent US 12,105,753
Granted Patent B2
US 12,105,753 · App. 17/946,915 · Granted Oct 1, 2024

Automated cover song identification

Inventors: Markus K. Cremer (Orinda, CA); Zafar Rafii (Berkeley, CA); Robert Coover (Orinda, CA); Prem Seetharaman (Chicago, IL)
Assignee: Gracenote, Inc.
G06F16/683G06Q50/184
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,105,753
App. No.
17/946,915
Granted
Oct 1, 2024
Kind
B2
Abstract

Example systems and methods for automated cover song identification are disclosed. An example apparatus includes at least one memory, machine-readable instructions, and one or more processors to execute the machine-readable instructions to at least execute a constant Q transform on time slices of first audio data to output constant Q transformed time slices, binarize the constant Q transformed time slices to output binarized and constant Q transformed time slices, execute a two-dimensional Fourier transform on time windows within the binarized and constant Q transformed time slices to output two-dimensional Fourier transforms of the time windows, generate a reference data structure based on a sequential order of the two-dimensional Fourier transforms, store the reference data structure in a database, and identify a query data structure associated with query audio data as a cover rendition of the audio data based on a comparison of the query and reference data structures.

Claims (73)

1. An apparatus comprising:

at least one memory;

machine-readable instructions; and

one or more processors to execute the machine-readable instructions to at least:

execute a constant Q transform on time slices of query audio data to output constant Q transformed time slices;

binarize the constant Q transformed time slices to output binarized and constant Q transformed time slices;

execute a two-dimensional Fourier transform on time windows within the binarized and constant Q transformed time slices to output two-dimensional Fourier transforms of the time windows;

generate a reference data structure based on a sequential order of the two-dimensional Fourier transforms;

store the reference data structure in a database; and

identify a query data structure associated with query audio data as a cover rendition of the audio data based on a comparison of the query data structure and the reference data structure using a similarity matrix, wherein the similarity matrix indicates at least one degree to which reference portions of the reference data structure are associated with query portions of the query data structure, and wherein the at least one degree satisfies a corresponding threshold.

2. The apparatus of claim 1 , wherein the one or more processors are to:

determine a distance measure between the query data structure and the reference data structure based on the similarity matrix; and

store an association in the database between the reference audio and the query audio based on the distance measure, the association to identify the query audio as the cover rendition.

3. The apparatus of claim 2 , wherein the one or more processors are to:

convolve the similarity matrix with a checkerboard kernel to generate a first convolved similarity matrix, the first convolved similarity matrix to include one or more positive elements and one or more negative elements; and

replace the one or more negative elements with zeros to generate a second convolved similarity matrix; and wherein:

the determination of the distance measure between the query data structure and the reference data structure is based on the second convolved similarity matrix.

4. The apparatus of claim 1 , wherein the one or more processors are to:

group the binarized and constant Q transformed time slices into the time windows prior to the execution of the two-dimensional Fourier transform on the time windows, the time windows to include overlapping time windows of uniform duration, and apply a blur algorithm to the two-dimensional Fourier transforms of the time windows prior to the sequential ordering of the two-dimensional Fourier transforms in the reference data structure.

5. The apparatus of claim 1 , wherein the one or more processors are to, for respective ones of the constant Q transformed time slices, determine a median value of a range of constant transformed time slices that encompasses the respective ones of the constant Q transformed time slices and binarizing the constant Q transformed time slices based on the median value.

6. The apparatus of claim 1 , wherein the one or more processors are to:

obtain metadata associated with the query audio from a content source;

access the database using the metadata to identify a plurality of reference data structures including the reference data structure;

determine a rank of the reference data structure with respect to the plurality of reference data structures; and

after a determination that the rank of the reference data structure satisfies a threshold, identify the reference data structure for comparison with the query data structure.

7. The apparatus of claim 6 , wherein the content source is at least one of (i) a stream of a live broadcast, (ii) a music sharing site, (iii) a video sharing site, or (iv) a social networking feed, a post, update, or a tweet of a social network.

8. A non-transitory computer readable storage medium comprising instructions that, when executed, cause one or more processors to at least:

execute a constant Q transform on time slices of query audio data to output constant Q transformed time slices;

binarize the constant Q transformed time slices to output binarized and constant Q transformed time slices;

execute a two-dimensional Fourier transform on time windows within the binarized and constant Q transformed time slices to output two-dimensional Fourier transforms of the time windows;

generate a reference data structure based on a sequential order of the two-dimensional Fourier transforms;

store the reference data structure in a database; and

identify a query data structure associated with query audio data as a cover rendition of the audio data based on a comparison of the query data structure and the reference data structure using a similarity matrix, wherein the similarity matrix indicates at least one degree to which reference portions of the reference data structure are associated with query portions of the query data structure, and wherein the at least one degree satisfies a corresponding threshold.

9. A non-transitory computer readable storage medium of claim 8 , wherein the instructions cause the one or more processors to:

determine a distance measure between the query data structure and the reference data structure based on the similarity matrix; and

store an association in the database between the reference audio and the query audio based on the distance measure, the association to identify the query audio as the cover rendition.

10. A non-transitory computer readable storage medium of claim 9 , wherein the instructions cause the one or more processors to:

convolve the similarity matrix with a checkerboard kernel to generate a first convolved similarity matrix, the first convolved similarity matrix to include one or more positive elements and one or more negative elements; and

replace the one or more negative elements with zeros to generate a second convolved similarity matrix; and wherein:

the determination of the distance measure between the query data structure and the reference data structure is based on the second convolved similarity matrix.

11. A non-transitory computer readable storage medium of claim 8 , wherein the instructions cause the one or more processors to:

group the binarized and constant Q transformed time slices into the time windows prior to the execution of the two-dimensional Fourier transform on the time windows, the time windows to include overlapping time windows of uniform duration; and

apply a blur algorithm to the two-dimensional Fourier transforms of the time windows prior to the sequential ordering of the two-dimensional Fourier transforms in the reference data structure.

12. A non-transitory computer readable storage medium of claim 8 , wherein the instructions cause the one or more processors to, for respective ones of the constant Q transformed time slices, determine a median value of a range of constant Q transformed time slices that encompasses the respective ones of the constant Q transformed time slices and binarizing the constant Q transformed time slices based on the median value.

13. A non-transitory computer readable storage medium of claim 8 , wherein the instructions cause the one or more processors to:

obtain metadata associated with the query audio from a content source;

access the database using the metadata to identify a plurality of reference data structures including the reference data structure;

determine a rank of the reference data structure with respect to the plurality of reference data structures; and

after a determination that the rank of the reference data structure satisfies a threshold, identify the reference data structure for comparison with the query data structure.

14. A non-transitory computer readable storage medium of claim 13 , wherein the content source is at least one of (i) a stream of a live broadcast, (ii) a music sharing site, (iii) a video sharing site, or (iv) a social networking feed, a post, update, or a tweet of a social network.

15. A method comprising:

executing a constant Q transform on time slices of query audio data to output constant Q transformed time slices;

binarizing the constant Q transformed time slices to output binarized and constant transformed time slices;

executing a two-dimensional Fourier transform on time windows within the binarized and constant Q transformed time slices to output two-dimensional Fourier transforms of the time windows;

generating a reference data structure based on a sequential order of the two-dimensional Fourier transforms;

storing the reference data structure in a database; and

identifying a query data structure associated with query audio data as a cover rendition of the audio data based on a comparison of the query data structure and the reference data structure using a similarity matrix, wherein the similarity matrix indicates at least one degree to which reference portions of the reference data structure are associated with query portions of the query data structure, and wherein the at least one degree satisfies a corresponding threshold.

16. The method of claim 15 , further including:

determining a distance measure between the query data structure and the reference data structure based on the similarity matrix; and

storing an association in the database between the reference audio and the query audio based on the distance measure, the association to identify the query audio as the cover rendition.

17. The method of claim 16 , further including:

convolving the similarity matrix with a checkerboard kernel to generate a first convolved similarity matrix, the first convolved similarity matrix to include one or more positive elements and one or more negative elements; and

replacing the one or more negative elements with zeros to generate a second convolved similarity matrix; and wherein:

the determining of the distance measure between the query data structure and the reference data structure is based on the second convolved similarity matrix.

18. The method of claim 15 , further including:

arranging the binarized and constant Q transformed time slices into the time windows prior to the execution of the two-dimensional Fourier transform on the time windows, the time windows to include overlapping time windows of uniform duration, and executing a blur algorithm to the two-dimensional Fourier transforms of the time windows prior to the sequential ordering of the two-dimensional Fourier transforms in the reference data structure.

19. The method of claim 15 , further including, for respective ones of the constant Q transformed time slices, determining a median value of a range of constant Q transformed time slices that encompasses the respective ones of the constant Q transformed time slices and binarizing the constant Q transformed time slices based on the median value.

20. The method of claim 15 , further including:

obtaining metadata associated with the audio from a content source;

querying the database using the metadata to identify a plurality of reference data structures including the reference data structure;

determining a rank of the reference data structure with respect to the plurality of reference data structures; and

after a determination that the rank of the reference data structure satisfies a threshold, identifying the reference data structure for comparison with the query data structure.

21. The method of claim 20 , wherein the content source is at least one of (i) a stream of a live broadcast, (ii) a music sharing site, (iii) a video sharing site, or (iv) a social networking feed, a post, update, or a tweet of a social.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 19, 2022
From: CREMER, MARKUS K.; RAFII, ZAFAR; COOVER, ROBERT; SEETHARAMAN, PREM
To: GRACENOTE, INC.
Reel/Frame 061135/0631 →
Continuity (4)
Continuation 17065479 · Oct 7, 2020
Continuation 15698557 · Sep 7, 2017
Provisional Application 62441469 · Jan 2, 2017
Related Publication 20230008776A1 · Jan 12, 2023