Audio identification based on data structure
Example systems and methods represent audio using a sequence of two-dimensional (2D) Fourier transforms (2DFTs), and such a sequence may be used by a specially configured machine to perform audio identification, such as for cover song identification. Such systems and methods are robust to timbral changes, time skews, and pitch skews. In particular, a special data structure provides a time-series representation of audio, and this time-series representation is robust to key changes, timbral changes, and small local tempo deviations. Accordingly, the systems and methods described herein analyze cross-similarity between these time-series representations. In some example embodiments, such systems and methods extract features from an audio fingerprint and calculate a distance measure that is robust and invariant to changes in musical structure.
1. A computerized method comprising:
accessing, using one or more hardware processors, reference audio to be represented by a reference data structure to be generated and stored in a reference database;
generating, using the one or more hardware processors, the reference data structure from the reference audio by at least:
performing a constant Q transform on multiple reference time slices of the reference audio;
binarizing the constant Q transformed reference time slices of the reference audio;
performing a two-dimensional Fourier transform on multiple reference time windows within the binarized and constant Q transformed reference time slices of the reference audio to obtain two-dimensional Fourier transforms of the reference time windows; and
sequentially ordering the two-dimensional Fourier transforms of the reference time windows in the reference data structure;
creating, within the reference database, a data association between the reference audio and the generated reference data structure that includes the sequentially ordered two-dimensional Fourier transforms of the reference time windows, the created data association indicating that the reference data structure is an identifier of the reference audio;
accessing query audio to be represented by a query data structure for comparison to the reference data structure that represents the reference audio; and
generating the query data structure from the query audio by at least:
performing a constant Q transform on multiple query time slices of the query audio;
binarizing the constant Q transformed query time slices of the query audio by, for each constant Q transformed query time slice, calculating a median value of a range of constant Q transformed query time slices that encompasses the constant Q transformed query time slice and binarizing the constant Q transformed query time slice based on the calculated median value of the range;
performing a two-dimensional Fourier transform on multiple query time windows within the binarized and constant Q transformed query time slices of the query audio to obtain two-dimensional Fourier transforms of the query time windows; and
sequentially ordering the two-dimensional Fourier transforms of the query time windows in the query data structure.
2. The computerized method of claim 1 , wherein:
binarizing the constant Q transformed reference time slices of the reference audio includes, for each respective constant Q transformed reference time slice, calculating a median value of a range of constant Q transformed reference time slices that encompasses the respective constant Q transformed reference time slice and binarizing the constant Q transformed reference time slices based on the calculated median value of the range.
3. The computerized method of claim 1 , further including:
grouping the binarized and constant Q transformed reference time slices of the reference audio into the multiple reference time windows prior to the performing of the two-dimensional Fourier transform on the multiple reference time windows, the multiple reference time windows including overlapping reference time windows of uniform duration.
4. The computerized method of claim 1 , wherein:
the generating of the reference data structure includes applying a blur algorithm to the two-dimensional Fourier transforms of the reference time windows prior to the sequential ordering of the two-dimensional Fourier transforms in the reference data structure.
5. The computerized method of claim 1 , further including:
grouping the binarized and constant Q transformed query time slices of the query audio into the multiple query time windows prior to the performing of the two-dimensional Fourier transform on the multiple query time windows, the query time windows including overlapping query time windows of uniform duration.
6. The computerized method of claim 1 , further including:
generating a similarity matrix that indicates degrees to which reference portions of the reference data structure are similar to query portions of the query data structure;
computing a distance measure between the query data structure and the reference data structure based on the generated similarity matrix; and
creating, within the reference database, a further data association between the reference audio and the query audio based on the computed distance measure between the query data structure and the reference data structure, the created further data association indicating that the query audio is a cover rendition of the reference audio.
7. The computerized method of claim 6 , further including:
convolving the generated similarity matrix with a checkerboard kernel, the convolved similarity matrix including positive elements and negative elements; and
replacing the negative elements of the convolved similarity matrix with zeros; and wherein:
the computing of the distance measure between the query data structure and the reference data structure is based on the convolved similarly similarity matrix with the negative elements replaced with zeros.
8. The computerized method of claim 7 , wherein:
the computing of the distance measure between the query data structure and the reference data structure includes:
identifying diagonals in the convolved similarity matrix with the negative elements replaced with zeros;
computing lengths and sums of the diagonals in the convolved similarity matrix;
computing multiplicative products of the lengths and the sums of the diagonals in the convolved similarity matrix;
ranking the diagonals based on the multiplicative products of the length and the sums of the diagonals in the convolved similarity matrix;
identifying a dominant subset of the diagonals based on the ranking of the diagonals in the convolved similarity matrix;
summing multiplicative products of lengths and sums of the dominant subset of the ranked diagonals in the convolved similarity matrix; and wherein:
the computing of the distance measure between the query data structure and the reference data structure is based on the summed multiplicative products of lengths and sums of the dominant subset of the ranked diagonals in the convolved similarity matrix.
9. The computerized method of claim 1 , further including:
receiving a request to identify the query audio, the request being received from a device; and
controlling the device by causing the device to present a notification that the query audio is a cover rendition of the reference audio based on a computed distance measure between the query data structure and the reference data structure.
10. A system comprising:
one or more hardware processors; and
a memory storing instructions that, when executed by one or more of the hardware processors, causes the system to perform operations including:
accessing, using the one or more hardware processors, reference audio to be represented by a reference data structure to be generated and stored in a reference database;
generating, using the one or more hardware processors, the reference data structure from the reference audio by at least:
performing a constant Q transform on multiple reference time slices of the reference audio;
binarizing the constant Q transformed reference time slices of the reference audio;
performing a two-dimensional Fourier transform on multiple reference time windows within the binarized and constant Q transformed reference time slices of the reference audio to obtain two-dimensional Fourier transforms of the reference time windows; and
sequentially ordering the two-dimensional Fourier transforms of the reference time windows in the reference data structure;
creating, within the reference database, a data association between the reference audio and the generated reference data structure that includes the sequentially ordered two-dimensional Fourier transforms of the reference time windows, the created data association indicating that the reference data structure is an identifier of the reference audio;
accessing query audio to be represented by a query data structure for comparison to the reference data structure that represents the reference audio; and
generating the query data structure from the query audio by at least:
performing a constant Q transform on multiple query time slices of the query audio;
binarizing the constant Q transformed query time slices of the query audio by, for each constant Q transformed query time slice, calculating a median value of a range of constant Q transformed query time slices that encompasses the constant Q transformed query time slice and binarizing the constant Q transformed query time slices based on the calculated median value of the range;
performing a two-dimensional Fourier transform on multiple query time windows within the binarized and constant Q transformed query time slices of the query audio to obtain two-dimensional Fourier transforms of the query time windows; and
sequentially ordering the two-dimensional Fourier transforms of the query time windows in the query data structure.
11. The system of claim 10 , wherein:
binarizing the constant Q transformed reference time slices of the reference audio includes, for each constant Q transformed reference time slice, calculating a median value of a range of constant Q transformed reference time slices that encompasses the constant Q transformed reference time slice and binarizing the constant Q transformed reference time slices based on the calculated median value of the range.
12. The system of claim 10 , further including:
grouping the binarized and constant Q transformed reference time slices of the reference audio into the multiple reference time windows prior to the performing of the two-dimensional Fourier transform on the multiple reference time windows, the multiple reference time windows including overlapping reference time windows of uniform duration.
13. The system of claim 10 , wherein:
the generating of the reference data structure includes applying a blur algorithm to the two-dimensional Fourier transforms of the reference time windows prior to the sequential ordering of the two-dimensional Fourier transforms in the reference data structure.
14. The system of claim 10 , further including:
grouping the binarized and constant Q transformed query time slices of the query audio into the multiple query time windows prior to the performing of the two-dimensional Fourier transform on the multiple query time windows, the query time windows including overlapping query time windows of uniform duration.
15. The system of claim 10 , further including:
generating a similarity matrix that indicates degrees to which reference portions of the reference data structure are similar to query portions of the query data structure;
computing a distance measure between the query data structure and the reference data structure based on the generated similarity matrix; and
creating, within the reference database, a further data association between the reference audio and the query audio based on the computed distance measure between the query data structure and the reference data structure, the created further data association indicating that the query audio is a cover rendition of the reference audio.
16. The system of claim 15 , further including:
convolving the generated similarity matrix with a checkerboard kernel, the convolved similarity matrix including positive elements and negative elements; and
replacing the negative elements of the convolved similarity matrix with zeros; and wherein:
the computing of the distance measure between the query data structure and the reference data structure is based on the convolved similarly similarity matrix with the negative elements replaced with zeros.
17. The system of claim 16 , wherein:
the computing of the distance measure between the query data structure and the reference data structure includes:
identifying diagonals in the convolved similarity matrix with the negative elements replaced with zeros;
computing lengths and sums of the diagonals in the convolved similarity matrix;
computing multiplicative products of the lengths and the sums of the diagonals in the convolved similarity matrix;
ranking the diagonals based on the multiplicative products of the length and the sums of the diagonals in the convolved similarity matrix;
identifying a dominant subset of the diagonals based on the ranking of the diagonals in the convolved similarity matrix;
summing multiplicative products of lengths and sums of the dominant subset of the ranked diagonals in the convolved similarity matrix; and wherein:
the computing of the distance measure between the query data structure and the reference data structure is based on the summed multiplicative products of lengths and sums of the dominant subset of the ranked diagonals in the convolved similarity matrix.
18. The system of claim 10 , further including:
receiving a request to identify the query audio, the request being received from a device; and
controlling the device by causing the device to present a notification that the query audio is a cover rendition of the reference audio based on the computed distance measure between the query data structure and the reference data structure.
19. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations including:
accessing, using one or more hardware processors, reference audio to be represented by a reference data structure to be generated and stored in a reference database;
generating, using the one or more hardware processors, the reference data structure from the reference audio by at least:
performing a constant Q transform on multiple reference time slices of the reference audio;
binarizing the constant Q transformed reference time slices of the reference audio;
performing a two-dimensional Fourier transform on multiple reference time windows within the binarized and constant Q transformed reference time slices of the reference audio to obtain two-dimensional Fourier transforms of the reference time windows; and
sequentially ordering the two-dimensional Fourier transforms of the reference time windows in the reference data structure;
creating, within the reference database, a data association between the reference audio and the generated reference data structure that includes the sequentially ordered two-dimensional Fourier transforms of the reference time windows, the created data association indicating that the reference data structure is an identifier of the reference audio;
accessing query audio to be represented by a query data structure for comparison to the reference data structure that represents the reference audio; and
generating the query data structure from the query audio by at least:
performing a constant Q transform on multiple query time slices of the query audio;
binarizing the constant Q transformed query time slices of the query audio by, for each constant Q transformed query time slice, calculating a median value of a range of constant Q transformed query time slices that encompasses the constant Q transformed query time slice and binarizing the constant Q transformed query time slices based on the calculated median value of the range;
performing a two-dimensional Fourier transform on multiple query time windows within the binarized and constant Q transformed query time slices of the query audio to obtain two-dimensional Fourier transforms of the query time windows; and
sequentially ordering the two-dimensional Fourier transforms of the query time windows in the query data structure.
20. The non-transitory machine-readable storage medium of claim 19 , wherein:
binarizing the constant Q transformed reference time slices of the reference audio includes, for each constant Q transformed reference time slice, calculating a median value of a range of constant Q transformed reference time slices that encompasses the constant Q transformed reference time slice and binarizing the constant Q transformed reference time slices based on the calculated median value of the range.
21. The non-transitory machine-readable storage medium of claim 19 , further including instructions that, when executed by the one or more processors of the machine, cause the machine to perform operations including:
grouping the binarized and constant Q transformed reference time slices of the reference audio into the multiple reference time windows prior to the performing of the two-dimensional Fourier transform on the multiple reference time windows, the multiple reference time windows including overlapping reference time windows of uniform duration.
22. The non-transitory machine-readable storage medium of claim 19 , wherein:
the generating of the reference data structure includes applying a blur algorithm to the two-dimensional Fourier transforms of the reference time windows prior to the sequential ordering of the two-dimensional Fourier transforms in the reference data structure.
23. The non-transitory machine-readable storage medium of claim 19 , further including instructions that, when executed by the one or more processors of the machine, cause the machine to perform operations including:
grouping the binarized and constant Q transformed query time slices of the query audio into the multiple query time windows prior to the performing of the two-dimensional Fourier transform on the multiple query time windows, the query time windows including overlapping query time windows of uniform duration.
24. The non-transitory machine-readable storage medium of claim 19 , further including instructions that, when executed by the one or more processors of the machine, cause the machine to perform operations including:
generating a similarity matrix that indicates degrees to which reference portions of the reference data structure are similar to query portions of the query data structure;
computing a distance measure between the query data structure and the reference data structure based on the generated similarity matrix; and
creating, within the reference database, a further data association between the reference audio and the query audio based on the computed distance measure between the query data structure and the reference data structure, the created further data association indicating that the query audio is a cover rendition of the reference audio.
25. The non-transitory machine-readable storage medium of claim 24 , further including instructions that, when executed by the one or more processors of the machine, cause the machine to perform operations including:
convolving the generated similarity matrix with a checkerboard kernel, the convolved similarity matrix including positive elements and negative elements; and
replacing the negative elements of the convolved similarity matrix with zeros; and wherein:
the computing of the distance measure between the query data structure and the reference data structure is based on the convolved similarity matrix with the negative elements replaced with zeros.
26. The non-transitory machine-readable storage medium of claim 25 , wherein:
the computing of the distance measure between the query data structure and the reference data structure includes:
identifying diagonals in the convolved similarity matrix with the negative elements replaced with zeros;
computing lengths and sums of the diagonals in the convolved similarity matrix;
computing multiplicative products of the lengths and the sums of the diagonals in the convolved similarity matrix;
ranking the diagonals based on the multiplicative products of the length and the sums of the diagonals in the convolved similarity matrix;
identifying a dominant subset of the diagonals based on the ranking of the diagonals in the convolved similarity matrix;
summing multiplicative products of lengths and sums of the dominant subset of the ranked diagonals in the convolved similarity matrix; and wherein:
the computing of the distance measure between the query data structure and the reference data structure is based on the summed multiplicative products of lengths and sums of the dominant subset of the ranked diagonals in the convolved similarity matrix.
27. The non-transitory machine-readable storage medium of claim 19 , further including instructions that, when executed by the one or more processors of the machine, cause the machine to perform operations including:
receiving a request to identify the query audio, the request being received from a device; and
controlling the device by causing the device to present a notification that the query audio is a cover rendition of the reference audio based on the computed distance measure between the query data structure and the reference data structure.
28. A non-transitory machine-readable storage medium comprising instructions that, when executed, cause at least one processor of at least one machine to at least:
generate a reference data structure from reference audio by at least:
performing a constant Q transform on multiple reference time slices of the reference audio;
binarizing the constant Q transformed reference time slices of the reference audio;
performing a two-dimensional Fourier transform on multiple reference time windows within the binarized and constant Q transformed reference time slices of the reference audio to obtain two-dimensional Fourier transforms of the reference time windows; and
sequentially ordering the two-dimensional Fourier transforms of the reference time windows in the reference data structure;
generating a similarity matrix that indicates degrees to which reference portions of the reference data structure are similar to query portions of a query data structure, the query data structure based on query audio;
computing a distance measure between the query data structure and the reference data structure based on the generated similarity matrix; and
creating, within a reference database, a data association between the reference audio and the query audio based on the computed distance measure between the query data structure and the reference data structure, the data association indicating that the query audio is a cover rendition of the reference audio.
29. The non-transitory machine-readable storage medium of claim 28 , wherein the instructions, when executed, cause the at least one processor to:
convolve the similarity matrix with a checkerboard kernel to generate a first convolved similarity matrix, the first convolved similarity matrix including positive elements and negative elements; and
replace the negative elements of the first convolved similarity matrix with zeros to generate a second convolved similarity matrix; and wherein:
the computing of the distance measure between the query data structure and the reference data structure is based on the second convolved similarity matrix.