Computer interface for polyphonic stringed instruments
An interface device is described that allows the audio signals from a polyphonic stringed instrument to be introduced into a personal computer environment for feature extraction and signal processing.
1. A computer-implemented method for processing audio signals for a stringed instrument, comprising:
receiving a serial data stream with a serial data interface of a computing device, the serial data stream encoding a plurality of digital audio signals, each digital audio signal representing one of a plurality of strings of the stringed instrument;
extracting the encoded digital audio signals from the serial data stream using the computing device;
processing each of the extracted digital audio signals with the computing device to detect string events for one or more of the plurality of strings of the stringed instrument, including conducting a time-domain analysis of each of the extracted digital audio signals to identify local maxima and minima associated with each of the extracted digital audio signals in the time domain, and identifying the string events with reference to the local maxima and minima; and
classifying each of the string events as corresponding to one of a plurality of performance techniques used by a musician to generate the corresponding string event, the plurality of performance techniques including picking, plucking, tapping, hammering on, and hammering off.
2. The method of claim 1 further comprising processing the extracted digital audio signals including one or more of dynamics processing, equalization, pitch-shifting, filtering, frequency modulation, amplitude modulation, delay, reverberation, distortion, wave shaping, driving wave tables, stimulating resonances, gating, or limiting.
3. The method of claim 1 further comprising processing each of the extracted digital audio signals using a plurality of processing modules that simultaneously process each extracted digital audio signal in both a time domain and a frequency domain.
4. The method of claim 1 further comprising recording each of the digital audio signals for subsequent processing.
5. The method of claim 1 further comprising generating graphical representations corresponding to each of the digital audio signals, the graphical representations representing one or more of pitch, dynamics, or timbre for the corresponding string of the string instrument.
6. The method of claim 1 further comprising controlling a special effect using information extracted from the serial digital stream.
7. The method of claim 1 further comprising identifying a beginning of a note with reference to one or more of the string events.
8. The method of claim 1 wherein classifying the string events is done using a neural network.
9. The method of claim 1 further comprising extracting a plurality of audio signal characteristics from the extracted digital audio signals, wherein processing of the extracted digital audio signals is done with reference to the audio signal characteristics.
10. The method of claim 9 wherein the audio signal characteristics include any of a continuous pitch of a fundamental harmonic of a string, an amplitude, a centroid, brightness, even/odd harmonic balance, noise, spectral shape, or complete spectrum.
11. The method of claim 1 further comprising generating a plurality of acoustic instrument messages with reference to the extracted digital audio signals, each acoustic instrument message summarizing spectral information corresponding to a particular one of the plurality of strings of the stringed instrument.
12. The method of claim 1 wherein conducting the time-domain analysis comprises identifying positive-polarity segments and negative-polarity segments of each extracted digital audio signal, classifying the positive-polarity segments into positive-polarity event candidates and negative-polarity segments into negative-polarity event candidates for each extracted digital audio signal, and classifying at least some of the positive-polarity and negative polarity event candidates for one or more of the extracted digital audio signals as the string events.
13. The method of claim 1 wherein processing each of the extracted digital audio signals comprises tracking a fundamental harmonic and one or more overtones independently.
14. The method of claim 13 further comprising determining a pitch of the fundamental harmonic.
15. The method of claim 14 wherein the pitch of the fundamental harmonic is determined with reference to fret scanning information.
16. The method of claim 13 further comprising determining an even/odd harmonic balance from the fundamental harmonic and the one or more overtones.
17. A computer program product for processing audio signals for a stringed instrument, the computer program product comprising at least one non-transitory computer-readable storage medium having computer program instructions stored therein configured to enable at least one computing device to:
receive a serial data stream with a serial data interface of a computing device, the serial data stream encoding a plurality of digital audio signals, each digital audio signal representing one of a plurality of strings of the stringed instrument;
extract the encoded digital audio signals from the serial data stream using the computing device;
process each of the extracted digital audio signals with the computing device to detect string events for one or more of the plurality of strings of the stringed instrument, including conducting a time-domain analysis of each of the extracted digital audio signals to identify local maxima and minima associated with each of the extracted digital audio signals in the time domain, and identifying the string events with reference to the local maxima and minima; and
classify each of the string events as corresponding to one of a plurality of performance techniques used by a musician to generate the corresponding string event, the plurality of performance techniques including picking, plucking, tapping, hammering on, and hammering off.
18. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to process the extracted digital audio signals using dynamics processing, equalization, pitch-shifting, filtering, frequency modulation, amplitude modulation, delay, reverberation, distortion, wave shaping, driving wave tables, stimulating resonances, gating, or limiting.
19. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to process each of the extracted digital audio signals using a plurality of processing modules that simultaneously process each extracted digital audio signal in both a time domain and a frequency domain.
20. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to record each of the digital audio signals for subsequent processing.
21. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to generate graphical representations corresponding to each of the digital audio signals, the graphical representations representing one or more of pitch, dynamics, or timbre for the corresponding string of the string instrument.
22. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to control a special effect using information extracted from the serial digital stream.
23. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to identify a beginning of a note with reference to one or more of the string events.
24. The computer program product of claim 17 wherein the computer program instructions are configured to enable the at least one computing device to classify the string events using a neural network.
25. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to extract a plurality of audio signal characteristics from the extracted digital audio signals, and process the extracted digital audio signals with reference to the audio signal characteristics.
26. The computer program product of claim 25 wherein the audio signal characteristics include any of a continuous pitch of a fundamental harmonic of a string, an amplitude, a centroid, brightness, even/odd harmonic balance, noise, spectral shape, or complete spectrum.
27. The computer program product of claim 17 wherein the computer program instructions are further configured to enable the at least one computing device to generate a plurality of acoustic instrument messages with reference to the extracted digital audio signals, each acoustic instrument message summarizing spectral information corresponding to a particular one of the plurality of strings of the stringed instrument.
28. The computer program product of claim 17 wherein the computer program instructions are configured to enable the at least one computing device to conduct the time-domain analysis by identifying positive-polarity segments and negative-polarity segments of each extracted digital audio signal, classifying the positive-polarity segments into positive-polarity event candidates and negative-polarity segments into negative-polarity event candidates for each extracted digital audio signal, and classifying at least some of the positive-polarity and negative polarity event candidates for one or more of the extracted digital audio signals as the string events.
29. The computer program product of claim 17 wherein the computer program instructions are configured to enable the at least one computing device to process each of the extracted digital audio signals by tracking a fundamental harmonic and one or more overtones independently.
30. The computer program product of claim 29 wherein the computer program instructions are further configured to enable the at least one computing device to determine a pitch of the fundamental harmonic.
31. The computer program product of claim 30 wherein the computer program instructions are configured to enable the at least one computing device to determine the pitch of the fundamental harmonic with reference to fret scanning information.
32. The computer program product of claim 29 wherein the computer program instructions are further configured to enable the at least one computing device to determine an even/odd harmonic balance from the fundamental harmonic and the one or more overtones.