IP Library Granted Patent US 11,907,288
Granted Patent B2
US 11,907,288 · App. 16/927,577 · Granted Feb 20, 2024

Audio identification based on data structure

Inventors: Zafar Rafii (Berkeley, CA); Prem Seetharaman (Chicago, IL)
Assignee: Gracenote, Inc.
G06F16/686G06F16/61G06F17/14G10L25/27G10L25/51
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,907,288
App. No.
16/927,577
Granted
Feb 20, 2024
Kind
B2
Abstract

Example systems and methods are audio identification based on data structure are disclosed. An example apparatus includes memory, and one or more processors to execute instructions to execute a constant Q transform on query time slices of query audio, binarize the constant Q transformed query time slices, execute a two-dimensional Fourier transform on query time windows within the binarized and constant Q transformed query time slices to generate two-dimensional Fourier transforms of the query time windows, sequentially order the two-dimensional Fourier transforms in a query data structure, and identify the query audio as a cover rendition of reference audio based on a comparison between the query data structure and a reference data structure associated with the reference audio.

Claims (119)

1. An apparatus comprising:

memory; and

one or more processors to execute instructions to:

execute a constant Q transform on query time slices of query audio;

binarize the constant Q transformed query time slices;

execute a two-dimensional Fourier transform on query time windows within the binarized constant Q transformed query time slices to generate two-dimensional Fourier transforms of the query time windows;

sequentially order the two-dimensional Fourier transforms in a query data structure; and

identify the query audio as a cover rendition of reference audio based on a comparison between the query data structure and a reference data structure associated with the reference audio.

2. The apparatus of claim 1 , wherein the constant Q transformed query time slices include a first constant Q transformed query time slice, and the one or more processors are to:

calculate a median value of a range of constant Q transformed query time slices that includes the first constant Q transformed query time slice; and

binarize the first constant Q transformed query time slice based on the median value of the range.

3. The apparatus of claim 1 , wherein the one or more processors are to:

execute a constant Q transform on reference time slices of the reference audio;

binarize the constant Q transformed reference time slices;

execute a two-dimensional Fourier transform on reference time windows within the binarized constant Q transformed reference time slices to generate two-dimensional Fourier transforms of the reference time windows; and

sequentially order the two-dimensional Fourier transforms of the reference time windows in the reference data structure.

4. The apparatus of claim 1 , wherein the one or more processors are to:

store an association in a database between the reference audio and the reference data structure, the association indicates that the reference data structure is an identifier of the reference audio;

receive a request to identify the query audio, the request received from a device; and

control the device by causing the device to present a notification that the query audio is the cover rendition, the notification including the identifier.

5. The apparatus of claim 1 , wherein the one or more processors are to:

generate a similarity matrix that indicates degrees to which reference portions of the reference data structure are similar to query portions of the query data structure;

compute a distance measure between the query data structure and the reference data structure based on the similarity matrix; and

store an association in a database between the reference audio and the query audio based on a computed distance measure, the association identifying the query audio as the cover rendition.

6. The apparatus of claim 5 , wherein the one or more processors are to:

convolve the similarity matrix with a checkerboard kernel to generate a first convolved similarity matrix, the first convolved similarity matrix including positive elements and negative elements; and

replace the negative elements with zeros to generate a second convolved similarity matrix; and wherein:

the computing of the distance measure between the query data structure and the reference data structure is based on the second convolved similarity matrix.

7. The apparatus of claim 1 , wherein the one or more processors are to:

generate a similarity matrix based on reference portions of the reference data structure with respect to query portions of the query data structure;

convolve the similarity matrix with a checkerboard kernel to generate a first convolved similarity matrix including negative elements;

replace respective ones of the negative elements with zeros to generate a second convolved similarity matrix;

identify diagonals with the negative elements replaced with zeros in the second convolved similarity matrix;

compute a first set of lengths and sums of the diagonals;

compute multiplicative products of the first set;

rank the diagonals based on the multiplicative products;

identify a dominant subset of the diagonals based on the ranking;

sum multiplicative products of a second set of lengths and sums of the dominant subset; and

compute a distance measure between the query data structure and the reference data structure based on the summed multiplicative products.

8. The apparatus of claim 1 , wherein the one or more processors are to group the binarized constant Q transformed query time slices of the query audio into the query time windows prior to the executing of the two-dimensional Fourier transform on the query time windows, the query time windows including overlapping query time windows of uniform duration.

9. The apparatus of claim 1 , wherein the one or more processors are to apply a blur algorithm to the two-dimensional Fourier transforms of the query time windows prior to the sequential ordering of the two-dimensional Fourier transforms in the query data structure.

10. A non-transitory machine-readable medium comprising instructions that, when executed, cause one or more processors to at least:

execute a constant Q transform on query time slices of query audio;

binarize the constant Q transformed query time slices;

execute a two-dimensional Fourier transform on query time windows within the binarized constant Q transformed query time slices to generate two-dimensional Fourier transforms of the query time windows;

sequentially order the two-dimensional Fourier transforms in a query data structure; and

identify the query audio as a cover rendition of reference audio based on a comparison between the query data structure and a reference data structure associated with the reference audio.

11. The non-transitory machine-readable medium of claim 10 , wherein the constant Q transformed query time slices include a first constant Q transformed query time slice, and the instructions, when executed, cause the one or more processors to:

calculate a median value of a range of constant Q transformed query time slices that includes the first constant Q transformed query time slice; and

binarize the first constant Q transformed query time slice based on the median value of the range.

12. The non-transitory machine-readable medium of claim 10 , wherein the instructions, when executed, cause the one or more processors to:

execute a constant Q transform on reference time slices of the reference audio;

binarize the constant Q transformed reference time slices;

execute a two-dimensional Fourier transform on reference time windows within the binarized constant Q transformed reference time slices to generate two-dimensional Fourier transforms of the reference time windows; and

sequentially order the two-dimensional Fourier transforms of the reference time windows in the reference data structure.

13. The non-transitory machine-readable medium of claim 10 , wherein the instructions, when executed, cause the one or more processors to:

store an association in a database between the reference audio and the reference data structure, the association indicates that the reference data structure is an identifier of the reference audio;

receive a request to identify the query audio, the request received from a device; and

control the device by causing the device to present a notification that the query audio is the cover rendition, the notification including the identifier.

14. The non-transitory machine-readable medium of claim 10 , wherein the instructions, when executed, cause the one or more processors to:

generate a similarity matrix that indicates degrees to which reference portions of the reference data structure are similar to query portions of the query data structure;

compute a distance measure between the query data structure and the reference data structure based on the similarity matrix; and

store an association in a database between the reference audio and the query audio based on a computed distance measure, the association identifying the query audio as the cover rendition.

15. The non-transitory machine-readable medium of claim 14 , wherein the instructions, when executed, cause the one or more processors to:

convolve the similarity matrix with a checkerboard kernel to generate a first convolved similarity matrix, the first convolved similarity matrix including positive elements and negative elements; and

replace the negative elements with zeros to generate a second convolved similarity matrix; and wherein:

the computing of the distance measure between the query data structure and the reference data structure is based on the second convolved similarity matrix.

16. The non-transitory machine-readable medium of claim 10 , wherein the instructions, when executed, cause the one or more processors to:

generate a similarity matrix based on reference portions of the reference data structure with respect to query portions of the query data structure;

convolve the similarity matrix with a checkerboard kernel to generate a first convolved similarity matrix including negative elements;

replace respective ones of the negative elements with zeros to generate a second convolved similarity matrix;

identify diagonals with the negative elements replaced with zeros in the second convolved similarity matrix;

compute a first set of lengths and sums of the diagonals;

compute multiplicative products of the first set;

rank the diagonals based on the multiplicative products;

identify a dominant subset of the diagonals based on the ranking;

sum multiplicative products of a second set of lengths and sums of the dominant subset; and

compute a distance measure between the query data structure and the reference data structure based on the summed multiplicative products.

17. The non-transitory machine-readable medium of claim 10 , wherein the instructions, when executed, cause the one or more processors to group the binarized constant Q transformed query time slices of the query audio into the query time windows prior to the executing of the two-dimensional Fourier transform on the query time windows, the query time windows including overlapping query time windows of uniform duration.

18. The non-transitory machine-readable medium of claim 10 , wherein the instructions, when executed, cause the one or more processors to apply a blur algorithm to the two-dimensional Fourier transforms of the query time windows prior to the sequential ordering of the two-dimensional Fourier transforms in the query data structure.

19. A method comprising:

executing, using one or more processors, a constant Q transform on query time slices of query audio;

binarizing, using the one or more processors, the constant Q transformed query time slices;

executing, using the one or more processors, a two-dimensional Fourier transform on query time windows within the binarized constant Q transformed query time slices to generate two-dimensional Fourier transforms of the query time windows;

sequentially ordering, using the one or more processors, the two-dimensional Fourier transforms in a query data structure; and

identifying, using the one or more processors, the query audio as a cover rendition of reference audio based on a comparison between the query data structure and a reference data structure associated with the reference audio.

20. The method of claim 19 , wherein the constant Q transformed query time slices include a first constant Q transformed query time slice, and binarizing the constant Q transformed query time slices includes:

calculating a median value of a range of constant Q transformed query time slices that includes the first constant Q transformed query time slice; and

binarizing the first constant Q transformed query time slice based on the median value of the range.

21. The method of claim 19 , further including:

executing a constant Q transform on reference time slices of the reference audio;

binarizing the constant Q transformed reference time slices;

executing a two-dimensional Fourier transform on reference time windows within the binarized constant Q transformed reference time slices to generate two-dimensional Fourier transforms of the reference time windows; and

sequentially ordering the two-dimensional Fourier transforms of the reference time windows in the reference data structure.

22. The method of claim 19 , further including:

storing an association in a database between the reference audio and the reference data structure, the association indicates that the reference data structure is an identifier of the reference audio;

receiving a request to identify the query audio, the request received from a device; and

controlling the device by causing the device to present a notification that the query audio is the cover rendition, the notification including the identifier.

23. The method of claim 19 , further including:

generating a similarity matrix that indicates degrees to which reference portions of the reference data structure are similar to query portions of the query data structure;

computing a distance measure between the query data structure and the reference data structure based on the similarity matrix; and

storing an association in a database between the reference audio and the query audio based on a computed distance measure, the association identifying the query audio as the cover rendition.

24. The method of claim 23 , further including:

convolving the similarity matrix with a checkerboard kernel to generate a first convolved similarity matrix, the first convolved similarity matrix including positive elements and negative elements; and

replacing the negative elements with zeros to generate a second convolved similarity matrix; and wherein:

the computing of the distance measure between the query data structure and the reference data structure is based on the second convolved similarity matrix.

25. The method of claim 19 , further including:

generating a similarity matrix based on reference portions of the reference data structure with respect to query portions of the query data structure;

convolving the similarity matrix with a checkerboard kernel to generate a first convolved similarity matrix including negative elements;

replacing respective ones of the negative elements with zeros to generate a second convolved similarity matrix;

identifying diagonals with the negative elements replaced with zeros in the second convolved similarity matrix;

calculating a first set of lengths and sums of the diagonals;

calculating multiplicative products of the first set;

ranking the diagonals based on the multiplicative products;

identifying a dominant subset of the diagonals based on the ranking;

summing multiplicative products of a second set of lengths and sums of the dominant subset; and

calculating a distance measure between the query data structure and the reference data structure based on the summed multiplicative products.

26. The method of claim 19 , further including grouping the binarized constant Q transformed query time slices of the query audio into the query time windows prior to the executing of the two-dimensional Fourier transform on the query time windows, the query time windows including overlapping query time windows of uniform duration.

27. The method of claim 19 , further including applying a blur algorithm to the two-dimensional Fourier transforms of the query time windows prior to the sequential ordering of the two-dimensional Fourier transforms in the query data structure.

Assignments (4)
SECURITY INTEREST Recorded May 8, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: ARES CAPITAL CORPORATION
Reel/Frame 063574/0632 →
SECURITY INTEREST Recorded Apr 28, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: CITIBANK, N.A.
Reel/Frame 063561/0381 →
SECURITY AGREEMENT Recorded Jan 31, 2023
From: GRACENOTE DIGITAL VENTURES, LLC; GRACENOTE MEDIA SERVICES, LLC; GRACENOTE, INC.; TNC (US) HOLDINGS, INC.; THE NIELSEN COMPANY (US), LLC
To: BANK OF AMERICA, N.A.
Reel/Frame 063560/0547 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 11, 2020
From: RAFII, ZAFAR; SEETHARAMAN, PREM
To: GRACENOTE, INC.
Reel/Frame 053461/0072 →
Continuity (3)
Continuation 15698532 · Sep 7, 2017
Provisional Application 62385574 · Sep 9, 2016
Related Publication 20200342024A1 · Oct 29, 2020