Audio classification for information retrieval using sparse features
Methods, systems, and apparatus, including computer programs encoded on computer storage media, are provided for using audio features to classify audio for information retrieval. In general, one aspect of the subject matter described in this specification can be embodied in methods that include the actions of generating a collection of auditory images, each auditory image being generated from respective audio files according to an auditory model; extracting sparse features from each auditory image in the collection to generate a sparse feature vector representing the corresponding audio file; and ranking the audio files in response to a query including one or more words using the sparse feature vectors and a matching function relating sparse feature vectors to words in the query.
1. A computer-implemented method comprising:
generating a collection of auditory images including a plurality of audio images from each of a plurality of audio files, each auditory image being generated from respective audio files according to an auditory model;
for each audio file, extracting sparse features from each auditory image generated for the audio file to generate a sparse feature vector representing the audio file, wherein extracting sparse features from each auditory image comprises:
dividing the auditory image into multiple sub-images,
applying a feature extractor to each sub-image to generate corresponding local sparse codes, and
combining the sparse codes from each sub-image to form a sparse vector for the auditory image; and
ranking the audio files in response to a query including one or more words using the sparse feature vectors and a matching function relating sparse feature vectors to words in the query.
2. The method of claim 1 , where the matching function is generated using a training collection of annotated audio files, and where generating the matching function includes:
receiving the collection of annotated audio files, each annotated audio file having an auditory image and one or more keywords associated with the content of the audio file;
generating a sparse feature vector for each audio file in the collection; and
training the matching function using the sparse feature vectors and the one or more keywords for the collection of annotated audio files to determine a matrix of weights matching sparse features and keywords.
3. The method of claim 1 , further comprising:
training the matching function using a passive-aggressive model using extracted audio features.
4. The method of claim 3 , where the training learns a matrix W representing a mapping between spars features and keywords such that F W (q k ,a k + )>F W (q k ,a k − ) for all k.
5. The method of claim 1 , where ranking the audio files further comprises:
scoring each query word relative to each sparse feature vector and combining the scores across words to rank audio files relative to the query.
6. The method of claim 5 , where scoring each query word includes calculating a dot product between a set of weights for that word and a representation of the audio file with a particular sparse feature vector.
7. The method of claim 1 , where the auditory model is a cochlear model that mimics the behavior of a cochlea.
8. The method of claim 1 , where the auditory image is a stabilized auditory image.
9. The method of claim 1 , where the auditory image is an auditory correlogram.
10. A computer-implemented method comprising:
receiving a text query, the query including one or more query terms;
retrieving a matching function that relates keywords and sparse feature vectors, each sparse feature vector being derived from a particular audio file, wherein each sparse feature vector is generated using sparse features extracted from a plurality of audio images, and wherein the sparse features represent abstract local structural properties of the auditory image;
identifying one or more keywords from the query terms;
identifying one or more audio files responsive to the query using the matching function; and
presenting search results identifying the one or more audio files.
11. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:
generating a collection of auditory images including a plurality of audio images from each of a plurality of audio files, each auditory image being generated from respective audio files according to an auditory model;
for each audio file, extracting sparse features from each auditory image generated for the audio file to generate a sparse feature vector representing the audio file, wherein extracting sparse features from each auditory image comprises:
dividing the auditory image into multiple sub-images,
applying a feature extractor to each sub-image to generate corresponding local sparse codes, and
combining the sparse codes from each sub-image to form a sparse vector for the auditory image; and
ranking the audio files in response to a query including one or more words using the sparse feature vectors and a matching function relating sparse feature vectors to words in the query.
12. The computer storage medium of claim 11 , where the matching function is generated using a training collection of annotated audio files, and where generating the matching function includes:
receiving the collection of annotated audio files, each annotated audio file having an auditory image and one or more keywords associated with the content of the audio file;
generating a sparse feature vector for each audio file in the collection; and
training the matching function using the sparse feature vectors and the one or more keywords for the collection of annotated audio files to determine a matrix of weights matching sparse features and keywords.
13. The computer storage medium of claim 11 , further comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations including:
training the matching function using a passive-aggressive model using extracted audio features.
14. The computer storage medium of claim 13 , where the training learns a matrix W representing a mapping between spars features and keywords such that F W (q k ,a k + )>F W (q k ,a k − ) for all k.
15. The computer storage medium of claim 11 , where ranking the audio files further comprises:
scoring each query word relative to each sparse feature vector and combining the scores across words to rank audio files relative to the query.
16. The computer storage medium of claim 15 , where scoring each query word includes calculating a dot product between a set of weights for that word and a representation of the audio file with a particular sparse feature vector.
17. The computer storage medium of claim 11 , where the auditory model is a cochlear model that mimics the behavior of a cochlea.
18. The computer storage medium of claim 11 , where the auditory image is a stabilized auditory image.
19. The computer storage medium of claim 11 , where the auditory image is an auditory correlogram.
20. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:
receiving a text query, the query including one or more query terms;
retrieving a matching function that relates keywords and sparse feature vectors, each sparse feature vector being derived from a particular audio file, wherein each sparse feature vector is generated using sparse features extracted from a plurality of audio images, and wherein the sparse features represent abstract local structural properties of the auditory image;
identifying one or more keywords from the query terms;
identifying one or more audio files responsive to the query using the matching function; and
presenting search results identifying the one or more audio files.
21. A system comprising:
one or more computers configured to perform operations including:
generating a collection of auditory images including a plurality of audio images from each of a plurality of audio files, each auditory image being generated from respective audio files according to an auditory model;
for each audio file, extracting sparse features from each auditory image generated for the audio file to generate a sparse feature vector representing the audio file, wherein extracting sparse features from each auditory image comprises:
dividing the auditory image into multiple sub-images,
applying a feature extractor to each sub-image to generate corresponding local sparse codes, and
combining the sparse codes from each sub-image to form a sparse vector for the auditory image; and
ranking the audio files in response to a query including one or more words using the sparse feature vectors and a matching function relating sparse feature vectors to words in the query.
22. The system of claim 21 , where the matching function is generated using a training collection of annotated audio files, and where generating the matching function includes:
receiving the collection of annotated audio files, each annotated audio file having an auditory image and one or more keywords associated with the content of the audio file;
generating a sparse feature vector for each audio file in the collection; and
training the matching function using the sparse feature vectors and the one or more keywords for the collection of annotated audio files to determine a matrix of weights matching sparse features and keywords.
23. The system of claim 21 , further configured to perform operations comprising:
training the matching function using a passive-aggressive model using extracted audio features.
24. The system of claim 23 , where the training learns a matrix W representing a mapping between spars features and keywords such that F W (q k ,a k + )>F W (q k ,a k − ) for all k.
25. The system of claim 21 , where ranking the audio files further comprises:
scoring each query word relative to each sparse feature vector and combining the scores across words to rank audio files relative to the query.
26. The system of claim 25 , where scoring each query word includes calculating a dot product between a set of weights for that word and a representation of the audio file with a particular sparse feature vector.
27. The system of claim 21 , where the auditory model is a cochlear model that mimics the behavior of a cochlea.
28. The system of claim 21 , where the auditory image is a stabilized auditory image.
29. The system of claim 21 , where the auditory image is an auditory correlogram.
30. A system comprising:
one or more computers configured to perform operations including:
receiving a text query, the query including one or more query terms;
retrieving a matching function that relates keywords and sparse feature vectors, each sparse feature vector being derived from a particular audio file, wherein each sparse feature vector is generated using sparse features extracted from a plurality of audio images, and wherein the sparse features represent abstract local structural properties of the auditory image;
identifying one or more keywords from the query terms;
identifying one or more audio files responsive to the query using the matching function; and
presenting search results identifying the one or more audio files.