Audio matching with supplemental semantic audio recognition and report generation
System, apparatus and method for determining semantic information from audio, where incoming audio is sampled and processed to extract audio features, including temporal, spectral, harmonic and rhythmic features. The extracted audio features are compared to stored audio templates that include ranges and/or values for certain features and are tagged for specific ranges and/or values. The semantic information may be associated with audio codes to determine changing characteristics of identified media during a time period.
1. A processor-based method for producing supplemental information for media containing an embedded audio code, the method comprising:
obtaining an audio code from a device during a first time period, the audio code representing a first characteristic of the media, wherein the audio code is read from an audio portion of the media;
obtaining first semantic audio signature data from the device during the first time period, the first semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media; and
associating the audio code of the first time period to a second time period when the processor determines that second semantic audio signature data of the second time period substantially matches the first semantic audio signature data for the first time period.
2. The method of claim 1 , wherein:
the temporal feature includes at least one of an amplitude, a power, or a zero crossing of at least some of the audio of the media;
the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity or a spectral inharmonicity of at least some of the audio of the media;
the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation or an overall pitch strength of at least some of the audio of the media; and
the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation or an average tempo for at least some of the audio of the media.
3. The method of claim 1 , wherein the audio code is decoded by transforming at least some of the audio from a time domain to a frequency domain.
4. The method of claim 1 , wherein the first semantic audio signature data is formed by transforming at least some of the audio from a time domain to a frequency domain.
5. An apparatus for producing supplemental information for media containing an audio code, the apparatus comprising:
a processor to:
obtain an audio code from a device during a first time period, the audio code representing a first characteristic of the media, wherein the audio code is read from an audio portion of the media;
obtain first semantic audio signature data from the device during the first time period, the first semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media; and
associate the audio code of the first time period to a second time period when second semantic audio signature data of the second time period substantially matches the first semantic audio signature data for the first time period.
6. The apparatus of claim 5 , wherein:
the temporal feature includes at least one of an amplitude, a power and a zero crossing of at least some of the audio of the media;
the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the audio of the media;
the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the audio of the media; and
the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation, or an average tempo for at least some of the audio of the media.
7. The apparatus of claim 5 , wherein the audio code is formed by transforming at least some of the audio from a time domain to a frequency domain.
8. The apparatus of claim 5 , wherein the semantic audio signature data is formed by transforming at least some of the audio from a time domain to a frequency domain.
9. A tangible computer readable storage device comprising instructions, that when executed, cause a computing device to at least:
obtain an audio code from a device during a first time period, the audio code representing a first characteristic of media;
obtain first semantic audio signature data from the device during the first time period, the first semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media; and
associate the audio code of the first time period to a second time period when second semantic audio signature data of the second time period substantially matches the first semantic audio signature data of the first time period.
10. The tangible computer readable storage device of claim 9 , wherein:
the temporal feature includes at least one of an amplitude, a power and a zero crossing of at least some audio of the media;
the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the audio of the media;
the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the audio of the media; and
the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation, or an average tempo for at least some of the audio of the media.
11. The tangible computer readable storage device of claim 9 , wherein the audio code is formed by transforming at least some of the audio from a time domain to a frequency domain.
12. The tangible computer readable storage device of claim 9 , wherein the semantic audio signature data is formed by transforming at least some of the audio from a time domain to a frequency domain.
13. A processor-based method for producing supplemental information for media containing audio codes, the method comprising:
obtaining audio codes from a device, the audio codes representing a first characteristic of the media, wherein the audio codes are read from audio portions of the media;
obtaining semantic audio signature data from the device, the semantic audio signature data includes at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media;
associating the semantic audio signature data to the audio codes; and
processing the associated semantic audio signature data and audio codes to determine a change in the second characteristic relative to the first characteristic.
14. The method of claim 13 , wherein:
the temporal feature includes at least one of an amplitude, a power, or a zero crossing of at least some of the audio of the media,
the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the audio of the media,
the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the audio of the media, and
the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation or an average tempo for at least some of the audio of the media.
15. The method of claim 13 , wherein the first characteristic includes one of a media identification, a media distributor identification, or a media broadcaster identification.
16. The method of claim 13 , wherein the second characteristic includes one of a genre, an instrumentation, a style, acoustical dynamics, or an emotive descriptor.
17. The method of claim 13 , wherein the audio codes are formed by transforming at least some of the media from a time domain to a frequency domain.
18. The method of claim 13 , wherein the semantic audio signature data is formed by transforming at least some of the media from a time domain to a frequency domain.
19. An apparatus for producing supplemental information for media containing audio codes, the apparatus comprising:
a processor to:
obtain audio codes from a device, the audio codes representing a first characteristic of the media, wherein the audio codes are read from audio portions of the media;
obtain semantic audio signature data from the device, the semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media;
associate the semantic audio signature data to the audio codes; and
process the associated semantic audio signature data and audio codes to determine a change in the second characteristic relative to the first characteristic.
20. The apparatus of claim 19 , wherein:
the temporal feature includes at least one of an amplitude, a power, or a zero crossing of at least some of the audio of the media,
the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the audio of the media,
the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the audio of the media, and
the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation or an average tempo for at least some of the audio of the media.
21. The apparatus of claim 19 , wherein the first characteristic includes one of a media identification, a media distributor identification, or a media broadcaster identification.
22. The apparatus of claim 19 , wherein the second characteristic includes one of a genre, an instrumentation, a style, acoustical dynamics, or an emotive descriptor.
23. The apparatus of claim 19 , wherein the audio codes are formed by transforming at least some of the media from a time domain to a frequency domain.
24. The apparatus of claim 19 , wherein the semantic audio signature data is formed by transforming at least some of the media from a time domain to a frequency domain.
25. A tangible computer readable storage device comprising instructions, that when executed, cause a computing device to at least:
obtain audio codes from a device, the audio codes representing a first characteristic of media;
obtain semantic audio signature data from the device, the semantic audio signature data being a measure of at least one of a temporal feature, a spectral feature, a harmonic feature, or a rhythmic feature relating to a second characteristic of the media;
associate the semantic audio signature data to the audio codes; and
process the associated semantic audio signature data and audio codes to determine a change in the second characteristic relative to the first characteristic.
26. The tangible computer readable storage device of claim 25 , wherein:
the temporal feature includes at least one of an amplitude, a power, or a zero crossing of at least some audio of the media,
the spectral feature includes at least one of a spectral centroid, a spectral rolloff, a spectral flux, a spectral flatness measure, a spectral crest factor, Mel-frequency cepstral coefficients, Daubechies wavelet coefficients, a spectral dissonance, a spectral irregularity, or a spectral inharmonicity of at least some of the audio of the media,
the harmonic feature includes at least one of a pitch, a tonality, a pitch class profile, harmonic changes, a main pitch class, an octave range of dominant pitch, a main tonal interval relation, or an overall pitch strength of at least some of the audio of the media, and
the rhythmic feature includes at least one of a rhythmic structure, a beat period, a rhythmic fluctuation or an average tempo for at least some of the audio of the media.
27. The tangible computer readable storage device of claim 25 , wherein the second characteristic includes one of a genre, an instrumentation, a style, acoustical dynamics, or an emotive descriptor.
28. The tangible computer readable storage device of claim 25 , wherein the audio codes are formed by transforming at least some of the media from a time domain to a frequency domain.
29. The tangible computer readable storage device of claim 25 , wherein the semantic audio signature data is formed by transforming at least some of the media from a time domain to a frequency domain.
30. The tangible computer readable storage device of claim 25 , wherein the first characteristic includes one of a media identification, a media distributor identification, or a media broadcaster identification.