Automatic labeling and control of audio algorithms by audio recognition
Controlling a multimedia software application using high-level metadata features and symbolic object labels derived from an audio source, wherein a first-pass of low-level signal analysis is performed, followed by a stage of statistical and perceptual processing, followed by a symbolic machine-learning or data-mining processing component is disclosed. This multi-stage analysis system delivers high-level metadata features, sound object identifiers, stream labels or other symbolic metadata to the application scripts or programs, which use the data to configure processing chains, or map it to other media. Embodiments of the invention can be incorporated into multimedia content players, musical instruments, recording studio equipment, installed and live sound equipment, broadcast equipment, metadata-generation applications, software-as-a-service applications, search engines, and mobile devices.
1. A non-transitory computer-readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for multi-stage audio signal analysis, the method comprising:
performing a first-stage processing operation on an audio signal, the first stage processing operation including a windowed signal analysis to calculate from the audio signal statistical descriptor features that are stored in a raw feature vector;
performing a second stage statistical processing operation on the raw feature vector to derive a reduced feature vector;
performing a third stage processing operation on the reduced feature vector to derive at least one sound object label that refers to the original audio signal; and
mapping the at least one sound object label into a stream of control events sent to a sound-object-driven, multimedia-aware software application, wherein the sound-object-driven multimedia-aware software application is responsive to the stream of control events to configure processing for the audio signal, and wherein any of the processing operations of the first through third stages are configurable.
2. The non-transitory computer-readable storage medium of claim 1 , wherein the audio signal is a file, and the method further comprises retrieving the file from a storage device.
3. The non-transitory computer-readable storage medium of claim 1 , wherein the audio signal is a stream, and the method further comprises receiving the stream as a digital signal from an input device or a network connection.
4. The non-transitory computer-readable storage medium of claim 1 , wherein the first stage processing operation is selected from the group consisting of amplitude-detection, FFT, MFCC, LPC, wavelet analysis, spectral measures, and stereo/spatial feature extraction.
5. The non-transitory computer-readable storage medium of claim 1 , wherein the second stage processing operation is selected from the group consisting of statistical averaging, mean/variance calculation, statistical moments, Gaussian mixture models, principal component analysis (PCA), independent subspace analysis (ISA), hidden Markhov models, tempo-tracking, pitch-tracking, peak/partial-tracking, onset detection, segmentation, and bark/sone mapping.
6. The non-transitory computer-readable storage medium of claim 1 , wherein the third stage processing operation is selected from the group consisting of support vector machines (SVN), neural networks (NN), partitioning/clustering, constraint satisfaction, stream labeling, rule-based expert systems, classification according to instrument, genre, or artist, musical transcription, and/or and sound object source separation.
7. The non-transitory computer-readable storage medium of claim 2 or 3 , wherein the audio signal is selected from the group consisting of a song, music loop, music clips, sound track, sound effects, and audio signals, and wherein the windowed signal analysis is performed on the audio signal.
8. The non-transitory computer-readable storage medium of claim 1 , wherein any of the first through third stages are stored in a database and may be retrieved for use in a subsequent analytical operation.
9. The non-transitory computer-readable storage medium of claim 1 , wherein the first through fourth stages are all processed in real-time for use in an on-the-fly analytical operation.
10. The non-transitory computer-readable storage medium of claim 1 , wherein the sound-object-driven, multimedia-aware software application is a mixing/recording application, and the sound object label automates the selection of signal processing tools and the setup of a signal chain in the mixing/recording application.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the automation of the signal processing tools and the setup of a signal chain includes the use of a look-up table.
12. The non-transitory computer-readable storage medium of claim 10 , wherein the automation of the signal processing tools and the setup of a signal chain includes the use of one or more rules.
13. The non-transitory computer-readable storage medium of claim 1 , wherein the sound-object-driven, multimedia-aware software application is a mixing/recording application, and the sound object label automates the selection of an audio parameter encoded in a preset of the mixing/recording application.
14. The non-transitory computer-readable storage medium of claim 1 , wherein the sound-object-driven, multimedia-aware software application adjusts in real-time an internal signal processing parameter of a mixing console in response to the at least one sound object label.
15. The non-transitory computer-readable storage medium of claim 1 , wherein the sound-object-driven, multimedia-aware software application is executable to select a codec in response to the type of audio source associated with the audio signal.
16. The non-transitory computer-readable storage medium of claim 1 , wherein the sound-object-driven, multimedia-aware software application is executable to allow processing of the audio signal when the audio signal corresponds to a particular sound type.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the sound-object-driven, multimedia-aware software application is executable to allow processing of the audio signal when the particular sound type exceeds a particular amplitude.
18. The non-transitory computer-readable storage medium of claim 1 , wherein the sound-object-driven, multimedia-aware software application is executable to allow processing of the audio signal when the audio signal exhibits a particular characteristic selected from the group consisting of feedback, distortion, pitch, and reverb.
19. The non-transitory computer-readable storage medium of claim 1 , wherein the sound-object-driven, multimedia-aware software application is executable to apply a descriptive label associated with semantic data derived from the audio signal.
20. The non-transitory computer-readable storage medium of claim 1 , wherein the sound-object-driven, multimedia-aware software application is executable to identify an environment surrounding a source of the audio signal.
21. The non-transitory computer-readable storage medium of claim 20 , wherein a functionality of a mobile device is configured in response to identification of the environment.
22. The non-transitory computer-readable storage medium of claim 21 , wherein the functionality of the mobile device includes the selection and volume of a ring tone.
23. The non-transitory computer readable storage medium of claim 1 , wherein the method is executable to identify a medically-relevant characteristic of the audio signal.
24. A method comprising:
performing a first-stage processing operation on an audio signal, the first stage processing operation including a windowed signal analysis that derives a raw feature vector;
performing a second stage statistical processing operation on the raw feature vector to derive a reduced feature vector;
performing a third stage processing operation on the reduced feature vector to derive at least one sound object label that refers to the original audio signal; and
mapping the at least one sound object label into a stream of control events sent to a sound-object-driven, multimedia-aware software application, wherein the sound-object-driven multimedia-aware software application is responsive to the stream of control events to configure processing for the audio signal, and wherein any of the processing operations of the first through third stages are configurable.
25. The method claim 24 , wherein the first through fourth stages are all processed in real-time for use in an on-the-fly analytical operation.
26. The method 24 , wherein the sound-object-driven, multimedia-aware software application is a mixing/recording application, and the sound object label automates the selection of signal processing tools and the setup of a signal chain in the mixing/recording application.
27. The method of claim 24 , wherein the sound-object-driven, multimedia-aware software application adjusts in real-time an internal signal processing parameter of a mixing console in response to the at least one sound object label.
28. The method claim 24 , wherein the sound-object-driven, multimedia-aware software application is executable to select a codec in response to the type of audio source associated with the audio signal.
29. The method of claim 24 , wherein the method is executable to identify an environment surrounding a source of the audio signal, and a functionality of a mobile device is configured in response to identification of the environment.
30. The method of claim 24 , wherein the sound-object-driven, multimedia-aware software application is executable in real-time to adjust an internal signal processing parameter of a mixing console in response to the audio sign al.
31. A method comprising:
performing a first-stage processing operation on an audio signal, the first stage processing operation including a windowed signal analysis that derives a raw feature vector;
performing a second stage statistical processing operation on the raw feature vector to derive a reduced feature vector;
performing a third stage processing operation on the reduced feature vector to derive at least one sound object label that refers to the original audio signal; and
mapping the at least one sound object label into a stream of control events sent to a sound-object-driven, multimedia-aware software application, wherein sound-object-driven, multimedia-aware software application adjusts an internal signal processing parameter of a mixing console in response to the at least one sound object label.