Intervalgram representation of audio for melody recognition
View Patent ↗A system, method, and computer readable storage medium generates an audio fingerprint for an input audio clip that is robust to differences in key, instrumentation, and other performance variations. The audio fingerprint comprises a sequence of intervalgrams that represent a melody in an audio clip according pitch intervals between different time points in the audio clip. The fingerprint for an input audio clip can be compared to a set of reference fingerprints in a reference database to determine a matching reference audio clip.
1. A computer-implemented method for matching audio clips, the method executed by a computer system, and comprising:
receiving an intervalgram representation for an input audio clip, the intervalgram representation comprising a sequence of intervalgram blocks, each intervalgram block associated with a different time period within the input audio clip, and each intervalgram block having a plurality of vectors, each vector of the intervalgram block representing a cross-correlation between a distribution of audio pitches occurring during a sub-period within the time period and a reference distribution associated with the time period, each distribution of audio pitches representing relative intensities of a plurality of different audio pitches present within a corresponding sub-period;
comparing, by a computing device, the intervalgram representation for the input audio clip to stored intervalgram representations corresponding to reference audio clips in a reference database;
selecting a reference audio clip from the reference database having an intervalgram representation best matching the intervalgram representation for the input audio clip; and
generating a recognition result indicative of the selected reference audio clip.
2. The method of claim 1 , further comprising, prior to receiving the intervalgram representation, generating the intervalgram representation, wherein generating the intervalgram representation comprises:
receiving an audio chromagram representing the input audio clip, the audio chromagram comprising a sequence of vectors, each vector in the audio chromagram corresponding to a different time point of the input audio clip, and each vector representing a distribution of audio pitches at the corresponding time point of the input audio clip; and
generating the intervalgram representation based on the audio chromagram.
3. The method of claim 2 , further comprising generating a chroma block from the audio chromagram, each chroma block corresponding to a different time sub-period, and each chroma block representing a weighted average of distributions of audio pitches within the corresponding time sub-period, wherein generating the chroma block comprises:
extracting a block of chroma vectors from the chromagram, the block of chroma vectors corresponding to one of the different time periods within the input audio clip; and
averaging the chroma vectors in the block of chroma vectors across time into a number of time bins, each time bin corresponding to a different time sub-period in the chroma block.
4. The method of claim 3 , wherein averaging the chroma vectors in the block of chroma vectors comprises applying a non-uniform sub-sampling such that time bins near the reference time are narrower than time bins at the edges of the block of chroma vectors.
5. The method of claim 2 , further comprising generating the audio chromagram, wherein generating the audio chromagram comprises:
receiving a pitchogram comprising a sequence of vectors, each vector corresponding to a different time point in the input audio clip, and each vector in the pitchogram representing a distribution of pitches at the time point in the input audio clip; and
quantizing and wrapping the distribution of pitches in of the sequence of vectors in the pitchogram into a number of chroma bins according to a weighted smoothing function, wherein each chroma bin represents a pitch independent of octave.
6. The method of claim 1 , further comprising generating the reference vector, wherein generating the reference vector comprises computing a weighted average of a plurality of chroma vectors within a limited time range surrounding a reference time, each chroma vector comprising a distribution of audio pitches at a occurring at a particular time point in the input audio clip.
7. The method of claim 6 , wherein computing the weighted average of the plurality of chroma vectors comprises applying a triangle weighting function to the plurality of chroma vectors.
8. The method of claim 1 , further comprising storing the intervalgram representation for the input audio clip to the reference database as an additional reference fingerprint.
9. A non-transitory computer readable storage medium storing computer-executable program instructions for matching audio clips, the program instructions when executed cause a processor to perform steps of:
receiving an intervalgram representation for an input audio clip, the intervalgram representation comprising a sequence of intervalgram blocks, each intervalgram block associated with a different time period within the input audio clip, and each intervalgram block having a plurality of vectors, each vector of the intervalgram block representing a cross-correlation between a distribution of audio pitches occurring at a sub-period within the time period and a reference distribution associated with the time period, each distribution of audio pitches representing relative intensities of a plurality of different audio pitches present within a corresponding sub-period;
comparing the intervalgram representation for the input audio clip to stored intervalgram representations corresponding to reference audio clips in a reference database;
selecting a reference audio clip from the reference database having an intervalgram representation best matching the intervalgram representation for the input audio clip; and
generating a recognition result indicative of the selected reference audio clip.
10. The non-transitory computer-readable storage medium of claim 9 , further comprising prior to receiving the intervalgram representation, generating the intervalgram representation, wherein generating the intervalgram representation comprises:
receiving an audio chromagram representing the input audio clip, the audio chromagram comprising a sequence of vectors, each vector in the audio chromagram corresponding to a different time point of the input audio clip, and each vector representing a distribution of audio pitches at the corresponding time point of the input audio clip; and
generating the intervalgram representation based on the audio chromagram.
11. The non-transitory computer-readable storage medium of claim 10 , wherein the program instructions further include instructions for generating a chroma block from the audio chromagram, each chroma block corresponding to a different time sub-period, and each chroma block representing a weighted average of distributions of audio pitches within the corresponding time sub-period, wherein generating the chroma block comprises:
extracting a block of chroma vectors from the chromagram, the block of chroma vectors corresponding to one of the different time periods within the input audio clip; and
averaging the chroma vectors in the block of chroma vectors across time into a number of time bins, each time bin corresponding to a different time sub-period in the chroma block.
12. The non-transitory computer-readable storage medium of claim 11 , wherein averaging the chroma vectors in the block of chroma vectors comprises applying a non-uniform sub-sampling such that time bins near the reference time are narrower than time bins at the edges of the block of chroma vectors.
13. The non-transitory computer-readable storage medium of claim 10 , wherein the program instructions further include instructions for generating the reference vector, wherein generating the reference vector comprises computing a weighted average of a plurality of chroma vectors within a limited time range surrounding a reference time, each chroma vector comprising a distribution of audio pitches at a occurring at a particular time point in the input audio clip.
14. The non-transitory computer-readable storage medium of claim 13 , wherein computing the weighted average of the plurality of chroma vectors comprises applying a triangle weighting function to the plurality of chroma vectors.
15. The non-transitory computer-readable storage medium of claim 10 , further comprising program instructions for generating the audio chromagram, wherein generating the audio chromagram comprises:
receiving a pitchogram comprising a sequence of vectors, each of the vectors corresponding to a different time point in the input audio clip, and each vector in the pitchogram representing a distribution of pitches at the time point in the input audio clip; and
quantizing and wrapping the distribution of pitches in of the sequence of vectors in the pitchogram into a number of chroma bins according to a weighted smoothing function, wherein each chroma bin represents a pitch independent of octave.
16. The non-transitory computer-readable storage medium 9 , further comprising storing the intervalgram representation for the input audio clip to the reference database as an additional reference fingerprint.