Pitch shift and time stretch resistant audio matching
View Patent ↗Systems and methods are provided herein relating to audio matching. Descriptors can be generated based on anchor points and interest points that characterize the local neighborhood surrounding the anchor point. Characterizing the local spectrogram neighborhood surrounding anchor points can be more robust to pitch shift distortions and time stretch distortions. Those anchor points surrounded by a lack of spectral activity or even spectral activity can be filtered from further examination. Using these pitch shift and time stretch resistant audio features within descriptors can provide for more accurate and efficient audio matching.
1. A system comprising:
a memory that has stored thereon computer executable components; and
a processor that executes the following computer executable components stored in the memory:
an input component that receives an audio sample;
a spectrogram component that generates a spectrogram of the audio sample;
an anchor detection component that generates a set of anchor points, wherein anchor points in the set of anchor points are local maxima in time/frequency windows of the spectrogram;
an interest point detection component that generates a set of interest points based on a defined strength threshold and a set of frequency bands;
a histogram component that, for each anchor point:
establish a frequency window around the anchor point,
identifies a predetermined number of closest interest points of the set of interest points to the anchor point within the frequency window,
divides the frequency window into a plurality of frequency bins,
determines respective counts of interest points of the closest interest points that lie within each frequency bin, and
generates a histogram for the anchor point based upon the respective counts of interest points of the closest interest points that lie within each frequency bin; and
a matching component that matches the audio sample to at least one reference audio sample based on comparing a set of descriptors generated from the histograms to sets of reference descriptors associated with a set of reference audio samples.
2. The system of claim 1 , further comprising:
a filter component that filters the histograms based on at least one of inactive frequency bins or equal frequency bin activity.
3. The system of claim 1 , further comprising:
a feature vector component that generates a set of feature vectors based on the histograms, wherein a feature vector in the set of feature vectors is associated with the anchor point and contains the respective counts of interest points of the closest interest points that lie within each frequency bin.
4. The system of claim 3 , further comprising:
a descriptor component that generates the set of descriptors for the audio sample by quantizing and hashing the set of feature vectors.
5. The system of claim 4 , wherein the descriptor component generates the set of descriptors further based on a time window and a time step.
6. The system of claim 4 , wherein the descriptor component hashes the set of feature vectors using a weighted minhash.
7. The system of claim 1 , wherein the defined strength threshold is a predetermined percentage of interest points with a highest strength, and the interest point detection component generates the set of interest points by selection of the predetermined percentage of interest points with the highest strength in each frequency band of the set of frequency bands.
8. The system of claim 7 , wherein respective sizes of the frequency bands increases as the associated frequencies increase.
9. The system of claim 1 , wherein the anchor point is at a center of the frequency window.
10. A method comprising:
receiving, by system including a processor, an audio sample;
generating, by the system, a spectrogram of the audio sample;
generating, by the system, a set of anchor points, wherein anchor points in the set of anchor points are local maxima in time/frequency windows of the spectrogram;
generating, by the system, a set of interest points based on a defined strength threshold and a set of frequency bands;
for each anchor point:
establishing, by the system, a frequency window around the anchor point,
identifying, by the system, a predetermined number of closest interest points of the set of interest points to the anchor point within the frequency window,
dividing, by the system, the frequency window into a plurality of frequency bins,
determining, by the system, respective counts of interest points of the closest interest points that lie within each frequency bin, and
generating, by the system, a histogram for the anchor point based upon the respective counts of interest points of the closest interest points that lie within each frequency bin; and
matching the audio sample to at least one reference audio sample based on comparing a set of descriptors generated from the histograms to sets of reference descriptors associated with a set of reference audio samples.
11. The method of claim 10 , the acts further comprising:
filtering, by the system, the histograms based on at least one of inactive frequency bins or equal frequency bin activity.
12. The method of claim 10 , the acts further comprising:
generating, by the system, a set of feature vectors based on the histograms, wherein a feature vector in the set of feature vectors is associated with the anchor point and contains the respective counts of interest points of the closest interest points that lie within each frequency bin.
13. The method of claim 12 , the acts further comprising:
quantizing, by the system, the set of feature vectors;
hashing, by the system, the set of feature vectors; and
generating, by the system, the set of descriptors for the audio sample based on the quantizing and the hashing.
14. The method of claim 13 , wherein the generating comprises generating the set of descriptors based on a time window and a time step.
15. The method of claim 13 , wherein the hashing comprises hashing the set of feature vectors using a weighted minhash.
16. The method of claim 13 , wherein the defined strength threshold is a predetermined percentage of interest points with a highest strength, and the generating the set of interest points comprises selecting the predetermined percentage of interest points with the highest strength in each frequency band of the set of frequency bands.
17. The method of claim 16 , wherein respective sizes of the frequency bands increases as the associated frequencies increase.
18. The method of claim 10 , wherein the anchor point is at a center of the frequency window.
19. A non-transitory computer-readable medium comprising computer-executable instructions that, in response to execution, cause a system including a processor to perform operations, comprising:
receiving an audio sample;
generating a spectrogram of the audio sample;
generating a set of anchor points, wherein anchor points in the set of anchor points are local maxima in time/frequency windows of the spectrogram;
generating a set of interest points based on a defined percentile for strength and a set of frequency bands;
for each anchor point:
establishing a frequency window around the anchor point,
identifying a predetermined number of closest interest points of the set of interest points to the anchor point within the frequency window,
dividing the frequency window into a plurality of frequency bins,
determining respective counts of interest points of the closest interest points that lie within each frequency bin, and
generating a histogram for the anchor point based upon the respective counts of interest points of the closest interest points that lie within each frequency bin; and
matching the audio sample to at least one reference audio sample based on comparing a set of descriptors generated from the histograms to sets of reference descriptors associated with a set of reference audio samples.
20. The non-transitory computer-readable medium of claim 19 , the operations further comprising:
generating a set of feature vectors based on the histograms, wherein a feature vector in the set of feature vectors is associated with the anchor point and contains the respective counts of interest points of the closest interest points that lie within each frequency bin;
quantizing the set of feature vectors;
hashing the set of feature vectors; and
generating the set of descriptors for the audio sample based on the quantizing and the hashing.