Context-dependent piano music transcription with convolutional sparse coding
The present disclosure presents a novel approach to automatic transcription of piano music in a context-dependent setting. Embodiments described herein may employ an efficient algorithm for convolutional sparse coding to approximate a music waveform as a summation of piano note waveforms convolved with associated temporal activations. The piano note waveforms may be pre-recorded for a particular piano that is to be transcribed and may optionally be pre-recorded in the specific environment where the piano performance is to be performed. During transcription, the note waveforms may be fixed and associated temporal activations may be estimated and post-processed to obtain the pitch and onset transcription. Experiments have shown that embodiments of the disclosure significantly outperform state-of-the-art music transcription methods trained in the same context-dependent setting, in both transcription accuracy and time precision, in various scenarios including synthetic, anechoic, noisy, and reverberant environments.
1. A method of transcribing a musical performance played on a piano, the method comprising:
generating a waveform dictionary for use with the piano playing the musical performance, the waveform dictionary being generated in a supervised manner by recording a plurality of waveforms in a non-transitory computer-readable storage medium, each of the plurality of waveforms being associated with a key of the piano;
recording the musical performance played on the piano;
determining a plurality of activation vectors associated with the recorded performance using the plurality of recorded waveforms, each of the plurality of activation vectors corresponding to a key of the piano and comprising one or more activations of the corresponding key over time by using a computer processor;
detecting local maxima from the plurality of activation vectors by using said computer processor;
inferring note onsets from the detected local maxima by using said computer processor;
outputting the inferred note onsets and the determined plurality of activation vectors by using said computer processor.
2. The method of claim 1 , wherein the plurality of recorded waveforms are associated with each individual piano note of the piano.
3. The method of claim 1 , wherein the plurality of recorded waveforms each have a duration of 0.5 second or more.
4. The method of claim 1 , wherein the plurality of activation vectors are determined using a convolutional sparse coding algorithm.
5. The method of claim 1 , wherein detecting local maxima from the plurality of activation vectors comprises discarding subsequent maxima following an initial local maxima that are within a predetermined time window.
6. The method of claim 5 , wherein the predetermined time window is at least 50 ms.
7. The method of claim 1 , wherein detecting local maxima from the plurality of activation vectors comprises discarding local maxima that are below a threshold that is associated with a highest peak in the plurality of activation vectors.
8. The method of claim 7 , wherein the threshold is 10% of the highest peak in the plurality of activation vectors such that local maxima that are 10% or less than the highest peak in the plurality of activation vectors are discarded.
9. A system for transcribing a musical performance played on a piano, the system comprising:
an audio recorder for recording a plurality of waveforms associated with keys of the piano and for recording the musical performance played on the piano;
a non-transitory computer-readable storage medium operably coupled with the audio recorder for storing the plurality of waveforms associated with keys of the piano to form a dictionary of elements and for storing the musical performance played on the piano;
a computer processor operably coupled with the non-transitory computer-readable storage medium and configured to:
determine a plurality of activation vectors associated with the stored performance using the plurality of stored waveform, each of the plurality of activation vectors corresponding to a key of the piano and comprising one or more activations of the corresponding key over time s;
detect local maxima from the plurality of activation vectors;
infer note onsets from the detected local maxima; and
output the inferred note onsets and the determined plurality of activation vectors.
10. The system of claim 9 , wherein the plurality of stored waveforms are associated with all individual piano notes of the piano.
11. The system of claim 9 , wherein the plurality of stored waveforms each have a duration of one second or more.
12. The system of claim 9 , wherein the plurality of activation vectors are determined by the computer processor using a convolutional sparse coding algorithm.
13. The system of claim 9 , wherein the computer processor detects local maxima from the plurality of activation vectors by discarding subsequent maxima following an initial local maxima that are within a predetermined time window.
14. The system of claim 13 , wherein the predetermined time window is at least 50 ms.
15. The system of claim 9 , wherein the computer processor detects local maxima from the plurality of activation vectors by discarding local maxima that are below a threshold that is associated with a highest peak in the plurality of activation vectors.
16. The system of claim 15 , wherein the threshold is 10% of the highest peak in the plurality of activation vectors such that local maxima that are 10% or less than the highest peak in the plurality of activation vectors are discarded.
17. A non-transitory computer-readable storage medium comprising a set of computer executable instructions for transcribing a musical performance played on an instrument, wherein execution of the instructions by a computer processor causes the computer processor to carry out the steps of:
generating a waveform dictionary for use with the piano playing the musical performance, the waveform dictionary being trained in a supervised manner by recording a plurality of waveforms in a non-transitory computer-readable storage medium, each of the plurality of waveforms being associated with a key of the instrument;
recording the musical performance played on the instrument;
determining a plurality of activation vectors associated with the recorded performance using the plurality of recorded waveforms, each of the plurality of activation vectors corresponding to a key of the piano and comprising one or more activations of the corresponding key over time;
detecting local maxima from the plurality of activation vectors;
inferring note onsets from the detected local maxima;
outputting the inferred note onsets and the determined plurality of activation vectors.
18. The non-transitory computer-readable storage medium of claim 17 , wherein the plurality of activation vectors are determined using a convolutional sparse coding algorithm.
19. The non-transitory computer-readable storage medium of claim 17 , wherein detecting local maxima from the plurality of activation vectors comprises discarding local maxima that are below a threshold that is associated with a highest peak in the plurality of activation vectors.
20. The non-transitory computer-readable storage medium of claim 17 , wherein detecting local maxima from the plurality of activation vectors comprises discarding subsequent maxima following an initial local maxima that are within a predetermined time window.