System and method for cluster-based audio event detection
Methods, systems, and apparatuses for audio event detection, where the determination of a type of sound data is made at the cluster level rather than at the frame level. The techniques provided are thus more robust to the local behavior of features of an audio signal or audio recording. The audio event detection is performed by using Gaussian mixture models (GMMs) to classify each cluster or by extracting an i-vector from each cluster. Each cluster may be classified based on an i-vector classification using a support vector machine or probabilistic linear discriminant analysis. The audio event detection significantly reduces potential smoothing error and avoids any dependency on accurate window-size tuning. Segmentation may be performed using a generalized likelihood ratio and a Bayesian information criterion, and the segments may be clustered using hierarchical agglomerative clustering. Audio frames may be clustered using K-means and GMMs.
1. A computer-implemented method comprising:
generating, by a computer, a plurality of audio frames partitioned from a plurality of audio signals;
generating, by the computer, a plurality of clusters, each cluster comprising one or more audio frames having similar features and associated with a type of sound;
for each cluster of the plurality of clusters, extracting, by the computer, a cluster-level feature vector based upon the similar features of the one or more audio frames of the cluster indicating the type of sound;
generating, by the computer, a plurality of incoming audio frames partitioned from an incoming audio signal;
generating, by the computer, a plurality of test clusters, each test cluster comprising one or more incoming audio frames having similar inbound features;
for each test cluster of the plurality of test clusters, extracting, by the computer, an inbound cluster-level feature vector based upon the similar inbound features of the one or more incoming audio frames of the test cluster; and
detecting, by the computer, the type of sound in each incoming audio frame based upon a similarity of the inbound cluster-level feature vector of the test cluster having the incoming audio frame to the cluster-level feature vector of the cluster associated with the type of sound.
2. The method according to claim 1 , further comprising:
for each of the audio frames partitioned from the plurality of audio signals:
generating, by the computer, a feature vector for the particular audio frame based upon a set of one or more features of the particular audio frame.
3. The method according to claim 1 , wherein generating each of the clusters comprises:
determining, by the computer, the type of sound associated with the particular cluster based upon a feature vector generated for each of the audio frames of the particular cluster.
4. The method according to claim 1 , wherein generating each of the clusters comprises:
determining, by the computer, the type of sound associated with the particular cluster using one or more supervised classifiers trained to classify one or more types of sound.
5. The method according to claim 4 , wherein the one or more supervised classifiers include a supervised multi-class classifier trained to classify a plurality of types of sound.
6. The method according to claim 1 , further comprising training, by the computer, a supervised classifier according to a label indicating one or more types of sound associated with a particular cluster.
7. The method according to claim 1 , wherein detecting the type of sound in each of the incoming audio frames comprises:
generating, by the computer, a feature vector for the particular incoming audio frame based upon a set of one or more features of the particular incoming audio frame.
8. The method according to claim 1 , wherein detecting the type of sound in the incoming audio frames comprises:
applying, by the computer, one or more supervised classifiers on one or more feature vectors generated for each of the incoming audio frames, wherein each of the supervised classifiers is trained to classify one or more types of sound.
9. The method according to claim 1 , further comprising:
generating, by the computer, a first output audio score by applying a first audio event detection component to a first type of feature of one or more incoming audio frames;
generating, by the computer, a second output audio score by applying a second audio event detection component on a second type of feature of the one or more incoming audio frames; and
generating, by the computer, a fused score for the one or more incoming audio frames based upon the first output audio score and the second output audio score, wherein the computer detects the type of sound in the one or more incoming audio frames based upon the fused score.
10. The method according to claim 1 , wherein a type of feature includes at least one of: Mel-Frequency Cepstral Coefficients, Perceptual Linear Prediction, and Relative Spectral Transform-Perceptual Linear Prediction.
11. A system comprising:
a non-transitory storage medium configured to store a plurality of audio signals; and
a processor configured to:
generate a plurality of audio frames partitioned from the plurality of audio signals;
generate a plurality of clusters, each cluster comprising one or more audio frames having similar features and associated with a type of sound;
for each cluster of the plurality of clusters, extract a cluster-level vector based upon the similar features of the one or more audio frames of the cluster indicating the type of sound;
generate a plurality of incoming audio frames partitioned from an incoming audio signal;
generate a plurality of test clusters, each test cluster comprises one or more incoming audio frames having similar inbound features;
for each test cluster of the plurality of test clusters, extract, by the computer, an inbound cluster-level feature vector based upon the similar inbound features of the one or more incoming audio frames of the test cluster; and
detect the type of sound in each incoming audio frame based upon a similarity of the inbound cluster-level feature vector of the test cluster having the incoming audio frame to the cluster-level feature vector of the cluster associated with the type of sound.
12. The system according to claim 11 , wherein the processor is further configured to:
for each of the audio frames partitioned from the plurality of audio signals:
generate a feature vector for the particular audio frame based upon a set of one or more features of the particular audio frame.
13. The system according to claim 11 , wherein the processor is further configured to:
determine the type of sound associated with the particular cluster based upon a feature vector generated for each of the audio frames of the particular cluster.
14. The system according to claim 11 , wherein the processor is further configured to:
determine the type of sound associated with the particular cluster using one or more supervised classifiers trained to classify one or more types of sound.
15. The system according to claim 14 , wherein the one or more supervised classifiers include a supervised multi-class classifier trained to classify a plurality of types of sound.
16. The system according to claim 11 , wherein the processor is further configured to:
train a supervised classifier according to a label indicating one or more types of sound associated with a particular cluster.
17. The system according to claim 11 , wherein the processor is further configured to:
generate a feature vector for the particular incoming audio frame based upon a set of one or more features of the particular incoming audio frame.
18. The system according to claim 11 , wherein the processor is further configured to:
apply one or more supervised classifiers on one or more feature vectors generated for each of the incoming audio frames, wherein each of the supervised classifiers is trained to classify one or more types of sound.
19. The system according to claim 11 , wherein the processor is further configured to:
generate a first output audio score by applying a first audio event detection component to a first type of feature of one or more incoming audio frames;
generate a second output audio score by applying a second audio event detection component on a second type of feature of the one or more incoming audio frames; and
generate a fused score for the one or more incoming audio frames based upon the first output audio score and the second output audio score, wherein the processor detects the type of sound in the one or more incoming audio frames based upon the fused score.
20. The system according to claim 11 , wherein a type of feature includes at least one of: Mel-Frequency Cepstral Coefficients, Perceptual Linear Prediction, and Relative Spectral Transform-Perceptual Linear Prediction.