Audio matching with semantic audio recognition and report generation
View Patent ↗System, apparatus and method for determining semantic information from audio, where incoming audio is sampled and processed to extract audio features, including temporal, spectral, harmonic and rhythmic features. The extracted audio features are compared to stored audio templates that include ranges and/or values for certain features and are tagged for specific ranges and/or values. The semantic information may be associated with audio signature data Extracted audio features that are most similar to one or more templates from the comparison are identified according to the tagged information. The tags are used to determine the semantic audio data that includes genre, instrumentation, style, acoustical dynamics, and emotive descriptor for the audio signal.
1. A processor-based method for producing supplemental information for audio signature data, comprising:
obtaining the audio signature data during a first time period, the audio signature data including data relating to at least one of time or frequency representing a first characteristic of media;
obtaining first semantic audio signature data for the first time period, the first semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media; and
storing in a memory, the audio signature data of the first time period in association with a second time period when the processor determines that second semantic audio signature data for the second time period substantially matches the first semantic audio signature data for the first time period.
2. The method of claim 1 , wherein the temporal feature includes at least one of amplitude, power, or zero crossing of at least some of the media.
3. The method of claim 1 , wherein the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, a Mel-frequency cepstral coefficient, a Daubechies wavelet coefficient, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the media.
4. The method of claim 1 , wherein the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, a harmonic change, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the media.
5. The method of claim 1 , wherein the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation, or an average tempo for at least some of the media.
6. The method of claim 1 , wherein the audio signature data is generated by transforming an audio signal of the media from a time domain to a frequency domain.
7. The method of claim 1 , wherein the semantic audio signature data is generated by transforming an audio signal of the media from a time domain to a frequency domain.
8. An apparatus for producing supplemental information for audio signature data, the apparatus including:
a processor to:
obtain the audio signature data during a first time period, the audio signature data including data relating to at least one of time or frequency representing a first characteristic of media;
obtain first semantic audio signature data for the first time period, the first semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media; and
a memory to store the audio signature data of the first time period in association with a second time period when the processor determines that second semantic audio signature data for the second time period substantially matches the first semantic audio signature data for the first time period.
9. The apparatus of claim 8 , wherein the temporal feature includes at least one of amplitude, power, or zero crossing of at least some of the media.
10. The apparatus of claim 8 , wherein the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, a Mel-frequency cepstral coefficient, a Daubechies wavelet coefficient, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the media.
11. The apparatus of claim 8 , wherein the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, a harmonic change, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the media.
12. The apparatus of claim 8 , wherein the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation, or an average tempo for at least some of the media.
13. The apparatus of claim 8 , wherein the audio signature data is generated by transforming an audio signal of the media from a time domain to a frequency domain.
14. The apparatus of claim 8 , wherein the semantic audio signature data is generated by transforming an audio signal of the media from a time domain to a frequency domain.
15. A processor-based method for producing supplemental information for audio signature data, comprising:
obtaining the audio signature data at an input from a data network, the audio signature data received from a device, the audio signature data including data relating to at least one of time or frequency representing a first characteristic of media;
obtaining semantic audio signature data at the input from the data network, the semantic audio signature data received from the device, the semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media;
associating the semantic audio signature data to the audio signature data using a processor; and
processing the associated semantic audio signature data and audio signature data to determine a change in the second characteristic relative to the first characteristic.
16. The method of claim 15 , wherein:
the temporal feature includes at least one of amplitude, power, or zero crossing of at least some of the media,
the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, a Mel-frequency cepstral coefficient, a Daubechies wavelet coefficient, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the media,
the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, a harmonic change, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the media, and
the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation, or an average tempo for at least some of the media.
17. The method of claim 15 , wherein the first characteristic includes at least one of a media identification, a media distributor identification, or a media broadcaster identification.
18. The method of claim 15 , wherein the second characteristic includes at least one of a genre, an instrumentation, a style, acoustical dynamics, or an emotive descriptor.
19. The method of claim 15 , wherein the audio signature data is generated by transforming at least some of the media from a time domain to a frequency domain.
20. The method of claim 15 , wherein the semantic audio signature data is generated by transforming at least some of the media from a time domain to a frequency domain.
21. An article of manufacture comprising instructions that, when executed, cause a processor to at least:
obtain audio signature data during a first time period, the audio signature data including data relating to at least one of time or frequency representing a first characteristic of media;
obtain first semantic audio signature data for the first time period, the first semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media; and
store in a memory, the audio signature data of the first time period in association with a second time period when the processor determines that second semantic audio signature data for the second time period substantially matches the first semantic audio signature data for the first time period.
22. The article of manufacture of claim 21 , wherein the temporal feature includes at least one of amplitude, power, or zero crossing of at least some of the media.
23. The article of manufacture of claim 21 , wherein the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, a Mel-frequency cepstral coefficient, a Daubechies wavelet coefficient, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the media.
24. The article of manufacture of claim 21 , wherein the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, a harmonic change, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the media.
25. The article of manufacture of claim 21 , wherein the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation, or an average tempo for at least some of the media.
26. The article of manufacture of claim 21 , wherein the audio signature data is generated by transforming an audio signal of the media from a time domain to a frequency domain.
27. The article of manufacture of claim 21 , wherein the semantic audio signature data is generated by transforming audio of the media from a time domain to a frequency domain.
28. An apparatus for producing supplemental information for audio signature data, the apparatus including:
a processor to:
obtain the audio signature data at an input from a data network, the audio signature data received from a device, the audio signature data including data relating to at least one of time or frequency representing a first characteristic of media;
obtain semantic audio signature data at the input from the data network, the semantic audio signature data received from the device, the semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media;
associate the semantic audio signature data to the audio signature data; and
process the associated semantic audio signature data and audio signature data to determine a change in the second characteristic relative to the first characteristic.
29. The apparatus of claim 28 , wherein:
the temporal feature includes at least one of amplitude, power, or zero crossing of at least some of the media,
the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, a Mel-frequency cepstral coefficient, a Daubechies wavelet coefficient, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the media,
the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, a harmonic change, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the media, and
the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation, or an average tempo for at least some of the media.
30. The apparatus of claim 28 , wherein the first characteristic comprises at least one of a media identification, a media distributor identification, or a media broadcaster identification.
31. The apparatus of claim 28 , wherein the second characteristic includes at least one of a genre, an instrumentation, a style, acoustical dynamics, or an emotive descriptor.
32. The apparatus of claim 28 , wherein the audio signature data is generated by transforming at least some of the media from a time domain to a frequency domain.
33. The apparatus of claim 28 , wherein the semantic audio signature data is generated by transforming at least some of the media from a time domain to a frequency domain.
34. An article of manufacture comprising instructions that, when executed, cause a processor to at least:
obtain the audio signature data at an input from a data network, the audio signature data received from a device, the audio signature data including data relating to at least one of time or frequency representing a first characteristic of media;
obtain semantic audio signature data at the input from the data network, the semantic audio signature data received from the device, the semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media;
associate the semantic audio signature data to the audio signature data; and
process the associated semantic audio signature data and audio signature data to determine a change in the second characteristic relative to the first characteristic.
35. The article of manufacture of claim 34 , wherein:
the temporal feature includes at least one of amplitude, power, or zero crossing of at least some of the media,
the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, a Mel-frequency cepstral coefficient, a Daubechies wavelet coefficient, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the media,
the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the media, and
the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation, or an average tempo for at least some of the media.
36. The article of manufacture of claim 34 , wherein the first characteristic includes at least one of a media identification, a media distributor identification, or a media broadcaster identification.
37. The article of manufacture of claim 34 , wherein the second characteristic includes at least one of a genre, an instrumentation, a style, acoustical dynamics, or an emotive descriptor.
38. The article of manufacture of claim 34 , wherein the audio signature data is generated by transforming an audio signal of the media from a time domain to a frequency domain.
39. The article of manufacture of claim 34 , wherein the semantic audio signature data is generated by transforming an audio signal of the media from a time domain to a frequency domain.