Audio matching based on harmonogram
View Patent ↗In an example context of identifying live audio, an audio processor machine accesses audio data that represents a query sound and creates a spectrogram from the audio data. Each segment of the spectrogram represents a different time slice in the query sound. For each time slice, the audio processor machine determines one or more dominant frequencies and an aggregate energy value that represents a combination of all the energy for that dominant frequency and its harmonics. The machine creates a harmonogram by representing these aggregate energy values at these dominant frequencies in each time slice. The harmonogram thus may represent the strongest harmonic components within the query sound. The machine can identify the query sound by comparing its harmonogram to other harmonograms of other sounds and may respond to a user's submission of the query sound by providing an identifier of the query sound to the user.
1. An apparatus comprising:
accessing means to access audio data that represents query sound to be identified;
first creating means to create a spectrogram of the audio data, different segments of the spectrogram representing amplitudes at frequencies in different time slices of the query sound;
determining means to determine a dominant frequency in a time slice of the query sound based on a segment of the spectrogram, the determining means including:
calculating means to calculate an aggregate energy value of a candidate frequency based on amplitudes of the candidate frequency and harmonics thereof represented in the segment of the spectrogram; and
identifying means to identify the candidate frequency as the dominant frequency based on the aggregate energy value of the candidate frequency being a largest aggregate energy value among aggregate energy values of frequencies whose amplitudes are represented in the segment of the spectrogram;
second creating means to create a query harmonogram of the audio data, different segments of the query harmonogram representing aggregate energy values of dominant frequencies in different time slices of the query sound; and
providing means to provide an identifier of the query sound based on a comparison of the query harmonogram to a reference harmonogram mapped to the identifier by a database.
2. The apparatus of claim 1 , wherein the second creating means is to indicate the aggregate energy value of the dominant frequency at a logarithm of the dominant frequency in a segment of the query harmonogram.
3. The apparatus of claim 1 , wherein the time slices of the query sound are between 10 and 30 milliseconds in duration, and the frequencies represented in the spectrogram each represent a quarter tone.
4. The apparatus of claim 1 , wherein the calculating means is to sum amplitudes of the candidate frequency and the harmonics thereof.
5. The apparatus of claim 1 , wherein the calculating means is to calculate weighted amplitudes of the candidate frequency and the harmonics thereof and sum the weighted amplitudes.
6. The apparatus of claim 1 , wherein the dominant frequency is a first most dominant frequency and indicates a strongest harmonic component of the time slice, and the determining means is to:
determine a second most dominant frequency that indicates a second strongest harmonic component of the time slice, the determining means including:
the calculating means to calculate an aggregate energy value of a further candidate frequency based on amplitudes of the further candidate frequency and harmonics thereof represented in the segment of the spectrogram; and
the identifying means to identify the further candidate frequency as the second most dominant frequency based on the aggregate energy value of the further candidate frequency being a second largest aggregate energy value among aggregate energy values of frequencies whose amplitudes are represented in the segment of the spectrogram; and
the second creating means is to create the query harmonogram by indicating the aggregate energy values of the first and second most dominant frequencies in a same segment of the query harmonogram.
7. The apparatus of claim 6 , wherein the second most dominant frequency and its harmonics are distinct from the first most dominant frequency and its harmonics in the segment of the spectrogram, and the calculating means is to calculate the aggregate energy value for the further candidate frequency not based on any amplitudes of the first most dominant frequency and harmonics thereof represented in the segment of the spectrogram.
8. The apparatus of claim 1 , further including:
generating means to generate a reference hash of the reference harmonogram;
indexing means to index the reference hash among multiple reference hashes of multiple reference harmonograms within the database; and
the generating means to generate a query hash of the query harmonogram based on the query harmonogram; and
the providing means is to provide the identifier of the query sound based on a determination that the query hash of the query harmonogram matches the indexed reference hash of the reference harmonogram.
9. The apparatus of claim 1 , further including:
removing means for removing the aggregate energy value of the dominant frequency from a segment of the query harmonogram based on a determination that a harmonic component indicated by the aggregate energy value is shorter than a threshold minimum duration within the query sound; and
the providing means is to provide the identifier of the query sound based on the query harmonogram with the aggregate energy value removed.
10. The apparatus of claim 1 , further including:
removing means for removing the aggregate energy value of the dominant frequency from a segment of the query harmonogram based on a determination that no adjacent segments of the query harmonogram contain aggregate energy values above a threshold minimum value within a threshold tolerance around the dominant frequency; and
the providing means is to provide the identifier of the query sound based on the query harmonogram with the aggregate energy value removed.
11. The apparatus of claim 1 , further including:
modifying means for modifying the query harmonogram by modifying the dominant frequency indicated by the aggregate energy value in a segment of the query harmonogram based on a neighboring aggregate energy value of a neighboring dominant frequency in a neighboring segment of the query harmonogram; and
the providing means is to provide the identifier of the query sound based on the modified query harmonogram.
12. The apparatus of claim 1 , further including:
modifying means for modifying the query harmonogram by generating an additional aggregate energy value of an additional dominant frequency in the segment of the query harmonogram based on a determination that an adjacent segment of the query harmonogram contains an aggregate energy value above a minimum threshold value but with no counterpart above a threshold minimum value within a threshold tolerance around the dominant frequency; and
the providing means is to provide the identifier of the query sound based on the modified query harmonogram.
13. The apparatus of claim 1 , wherein the identifying means are first identifying means, and further including:
second identifying means for identifying a query pattern of energy peaks in the query harmonogram based on a threshold minimum energy value; and
the providing means is to compare the query harmonogram to the reference harmonogram by comparing the query pattern of energy peaks to a reference pattern of energy peaks identified in the reference harmonogram.
14. The apparatus of claim 13 , wherein:
the second identifying means is to identify the query pattern of energy peaks in the query harmonogram based on a threshold minimum separation between peaks.
15. The apparatus of claim 1 , wherein:
the first creating means is to create the spectrogram of the audio data by creating a first two-dimensional array of amplitudes at frequencies over the time slices of the query sound, the created spectrogram being stored in the database as a first image; and
the second creating means is to create the query harmonogram of the audio data by creating a second two-dimensional array of aggregate energy values at frequencies over the time slices of the query sound, the created query harmonogram being stored in the database as a second image.
16. The apparatus of claim 1 , wherein:
the first creating means is to create the spectrogram of the audio data by performing a Constant Q Transform (CQT) of the audio data.
17. A non-transitory machine-readable storage medium comprising instructions that, when executed, cause a machine to at least:
create a spectrogram of audio data that represents query sound to be identified;
determine a dominant frequency in a time slice of the query sound based on a segment of the spectrogram by:
calculating an aggregate energy value of a candidate frequency based on amplitudes and harmonics of the candidate frequency represented in the segment of the spectrogram; and
identifying the candidate frequency as the dominant frequency based on the aggregate energy value having a greatest aggregate energy value of the aggregate energy values of frequencies whose amplitudes are represented in the segment of the spectrogram;
create a query harmonogram of the audio data; and
provide an identifier of the query sound based on a comparison of the query harmonogram to a reference harmonogram mapped to the identifier by a database.
18. The non-transitory machine-readable storage medium of claim 17 , wherein different segments of the spectrogram represent amplitudes at frequencies in different time slices of the query sound, and different segments of the query harmonogram represent aggregate energy values of dominant frequencies in different time slices of the query sound.
19. An apparatus comprising:
at least one processor; and
memory including instructions that, when executed, cause the at least one processor to:
create a spectrogram of audio data that represents query sound to be identified;
determine a dominant frequency in a time slice of the query sound based on a segment of the spectrogram by:
calculating an aggregate energy value of a candidate frequency based on amplitudes and harmonics of the candidate frequency represented in the segment of the spectrogram; and
identifying the candidate frequency as the dominant frequency based on the aggregate energy value having a greatest aggregate energy value of the aggregate energy values of frequencies whose amplitudes are represented in the segment of the spectrogram;
create a query harmonogram of the audio data; and
provide an identifier of the query sound based on a comparison of the query harmonogram to a reference harmonogram mapped to the identifier by a database.
20. The apparatus of claim 19 , wherein different segments of the spectrogram represent amplitudes at frequencies in different time slices of the query sound, and different segments of the query harmonogram represent aggregate energy values of dominant frequencies in different time slices of the query sound.