Music classifier and related methods
An audio device that includes a music classifier that determines when music is present in an audio signal is disclosed. The audio device is configured to receive audio, process the received audio, and to output the processed audio to a user. The processing may be adjusted based on the output of the music classifier. The music classifier utilizes a plurality of decision making units, each operating on the received audio independently. The decision making units are simplified to reduce the processing, and therefore the power, necessary for operation. Accordingly each decision making unit may be insufficient to determine music alone but in combination may accurately detect music while consuming power at a rate that is suitable for a mobile device, such as a hearing aid.
1. A music classifier for an audio device, the music classifier comprising:
a signal conditioning unit configured to transform a digitized, time-domain audio signal into a corresponding frequency domain signal including a plurality of frequency bands;
a plurality of decision making units operating in parallel that are each configured to evaluate one or more of the plurality of frequency bands to determine a plurality of feature scores, each feature score corresponding to a characteristic associated with music, the plurality of decision making units including:
a modulation activity tracking unit configured to output a feature score for modulation activity based on a ratio of a first value of an averaged wideband energy of the plurality of frequency bands to a second value of the averaged wideband energy of the plurality of frequency bands; and
a tone detection unit configured to output feature scores for tone in each frequency band based on (i) an amount of energy in the frequency band and (ii) a variance of the energy in the frequency band based on a first order differentiation; and
a combination and music detection unit configured to:
asynchronously receive feature scores from the plurality of decision making units, the decision making units configured to output feature scores at different intervals; and
combine the plurality of feature scores over a period of time to determine if the audio signal includes music.
2. The music classifier for the audio device according to claim 1 , wherein the plurality of decision making units include a beat detection unit.
3. The music classifier for the audio device according to claim 2 , wherein the beat detection unit is configured to detect, based on a correlation, a repeating beat pattern in a first frequency band that is the lowest of the plurality of frequency bands.
4. The music classifier for the audio device according to claim 2 , wherein the beat detection unit is configured to detect a repeating beat pattern, based on an output of a beat detection (BD) neural network.
5. The music classifier for the audio device according to claim 4 , wherein the beat detection unit is configured to select one or more frequency bands from the plurality of frequency bands and is configured to extract a plurality of features from each selected frequency band.
6. The music classifier for the audio device according to claim 5 , wherein the plurality of features extracted from each selected frequency band form a feature set including an energy mean, an energy standard deviation, an energy maximum, an energy kurtosis, an energy skewness, and an energy cross-correlation vector.
7. The music classifier for the audio device according to claim 6 , wherein the BD neural network receives the feature set for each selected band as a plurality of inputs.
8. The music classifier for the audio device according to claim 1 , wherein the second value corresponds a minimum of the averaged wideband energy and the first value corresponds to a maximum of the averaged wideband energy, the averaged wideband energy corresponding to an average of a sum of the energy in each of the plurality of frequency bands.
9. The music classifier for the audio device according to claim 1 , wherein the combination and music detection unit is configured to apply a weight to each feature score to obtain weighted feature scores and to sum the weighted feature scores to obtain a music score, each weight having a value that depends, in part, on the interval that the corresponding feature score is output from the decision making unit.
10. The music classifier for the audio device according to claim 9 , wherein the combination and music detection unit is further configured to accumulate music scores for a plurality of frames, to compute an average of the music scores for the plurality of frames, and to compare the average to a threshold.
11. The music classifier for the audio device according to claim 10 , wherein the combination and music detection unit is further configured to apply a hysteresis control to a music or no music output of the threshold.
12. A method for music detection in an audio signal, the method comprising:
receiving an audio signal;
digitizing the audio signal to obtain a digitized audio signal;
transforming the digitized audio signal into a plurality of frequency bands;
applying the plurality of frequency bands to a plurality of decision making units operating in parallel, the plurality of decision making units including:
a modulation activity tracking unit configured to output a feature score for modulation activity based on a ratio of a first value of an averaged wideband energy of the plurality of frequency bands to a second value of the averaged wideband energy of the plurality of frequency bands; and
a tone detection unit configured to output feature scores for tone in each frequency band based on (i) an amount of energy in the frequency band and (ii) a variance of the energy in the frequency band based on a first order differentiation; and
obtaining, asynchronously, a feature score from each of the plurality of decision making units, the decision making units configured to output feature scores at different intervals, and the feature score from each decision making unit corresponding to a probability that a particular music characteristic is included in the audio signal; and
combining the feature scores to detect music in the audio signal.
13. The method for music detection according to claim 12 , wherein the decision making units include a beat detection unit, and wherein:
obtaining a feature score from the beat detection unit includes:
detecting, based on a correlation, a repeating beat pattern in a first frequency band that is the lowest of the plurality of frequency bands.
14. The method for music detection according to claim 12 , wherein the decision making units include a beat detection unit, and wherein:
obtaining a feature score from the beat detection unit includes:
detecting, based on a neural network, a repeating beat pattern in the plurality of frequency bands.
15. The method for music detection according to claim 12 , wherein:
obtaining a feature score from the modulation activity tracking unit includes:
tracking a minimum averaged energy of a sum of the plurality of frequency bands as the second value and a maximum averaged energy of the sum of the plurality of frequency bands as the first value.
16. The method for music detection according to claim 12 , wherein the combining comprises;
multiplying the feature score from each of the plurality of decision making units with a respective weight to obtain a weighted score from each of the plurality of decision making units, each weight having a value that depends, in part, on the interval that the corresponding feature score is output from the decision making unit;
summing the weighted scores from the plurality of decision making units to obtain a music score;
accumulating music scores over a plurality of frames of the audio signal;
averaging the music scores from the plurality of frames of the audio signal to obtain an average music score; and
comparing the average music score to a threshold to detecting music in the audio signal.
17. The method for music detection in an audio signal according to claim 12 , further comprising:
modifying the audio signal based on the music detection; and
transmitting the audio signal.
18. A hearing aid, comprising:
a signal conditioning stage configured to convert a digitized audio signal to a plurality of frequency bands; and
a music classifier coupled to the signal conditioning stage, the music classifier including:
a feature detection and tracking unit that includes a plurality of decision making units operating in parallel, each decision making unit configured to generate a feature score corresponding to a probability that a particular music characteristic is included in the audio signal, the plurality of decision making units including:
a modulation activity tracking unit, the modulation activity tracking unit configured to output a feature score for modulation activity based on a ratio of a first value of an averaged wideband energy of the plurality of frequency bands to a second value of the averaged wideband energy of the plurality of frequency bands; and
a tone detection unit configured to output feature scores for tone in each frequency band based on (i) an amount of energy in the frequency band and (ii) a variance of the energy in the frequency band based on a first order differentiation; and
a combination and music detection unit configured to:
asynchronously receive feature scores from the plurality of decision making units, the decision making units configured to output feature scores at different intervals; and
combine the plurality of feature scores over time to detect music in the audio signal, the combination and music detection unit configured to produce a first signal indicating music while music is detected in the audio signal and configured to produce a second signal indicating no-music signal otherwise.
19. The hearing aid according to claim 18 , wherein the hearing aid includes an audio signal modifying stage coupled to the signal conditioning stage and to the music classifier, the audio signal modifying stage configured to process the plurality of frequency bands differently when a music signal is received than when a no-music signal is received.