Pitch selection modules in a system for automatic transcription of sung or hummed melodies
View Patent ↗The technology disclosed relates to audio signal processing. It includes a series of modules that individually are useful to solve audio signal processing problems. Among the problems addressed are buzz removal, selecting a pitch candidate among pitch candidates based on local continuity of pitch and regional octave consistency, making small adjustments in pitch, ensuring that a selected pitch is consistent with harmonic peaks, determining whether a given frame or region of frames includes harmonic, voiced signal, extracting harmonics from voice signals and detecting vibrato. One environment in which these modules are useful is transcribing singing or humming into a symbolic melody. Another environment that would usefully employ some of these modules is speech processing. Some of the modules, such as buzz removal, are useful in many other environments as well.
1. A method of selecting a pitch class for a frame among a sequence of frames that represent an audio signal, the method including:
processing electronically a sequence of frames that include at least one pitch estimate per frame;
transforming the pitch estimates for the frames into pitch class estimates that assign equal pitch classes to pitches in different octaves that have equal positions within their respective octaves;
constructing at least one pitch class consistency streak including pitch class estimates selected from consecutive frames that have pitch class estimates within a predetermined pitch class margin of one another; and
outputting data regarding pitch content of the frames based on at least the pitch classes in the pitch class consistency streak.
2. The method of claim 1 , further including octave selection, after the selecting the estimated pitch class, including:
defining an octave selection buffer that includes all or some of the frames in the pitch class consistency streak;
selecting an octave-wide band based on analysis of pitches in the frames in the octave selection buffer; and
assigning the estimated pitch classes of the frames in the octave selection buffer to pitches in the octave wide band, producing selected pitches.
3. The method of claim 2 , after assigning the pitch class estimates to the selected octave, further including applying a smoother to the selected pitches.
4. The method of claim 2 , after assigning the pitch class estimates to the selected octave, adjusting the selected pitches by no more than a predetermined adjustment band, including:
electronically processing spectrogram data for the frame in the sequence of frames, determining whether the selected pitch for the frame should be adjusted within a predetermined adjustment band to increase consistency between the selected pitch and frequencies of harmonic peaks in the spectrogram data.
5. The method of claim 2 , after assigning the pitch class estimates to the selected octave, adjusting the selected pitches by no more than a predetermined adjustment band, including:
electronically processing spectrogram data for the frame in the sequence of frames, determining whether the selected pitch for the frame should be adjusted within a predetermined adjustment band to increase consistency between the selected pitch and frequencies of harmonic peaks in the spectrogram data; and
using the adjusted selected pitch, searching the spectrogram data for the frame to find any additional harmonic peaks in the spectrogram data that had been missed in earlier processing and using all of the harmonic peaks relevant to the adjusted selected pitch, repeating the adjusting of the selected pitch.
6. An electronic signal processing component for selecting a pitch in frames that represent an audio signal, the component including:
an input port adapted to receive a stream of data frames including at least one pitch estimate per frame;
a modulo conversion processor coupled to the input port that assigns equal pitch classes to pitches in different octaves that have equal positions within their respective octaves;
a streak constructor processor coupled to receive the assigned pitch classes from the modulo conversion component, and to assign the frames to one or more pitch class consistency streaks of consecutive frames that have pitch class estimates within a predetermined pitch class margin of one another; and
an output port coupled to the streak constructor that outputs data regarding pitch content of the frames based on at least the pitch classes in the pitch class consistency streaks.
7. An electronic signal processing component for pitch determination, including the component of claim 6 , further including:
an octave assignment component coupled between the streak constructor and the output port, comprising
an octave selection buffer that includes all or some of the frames in the pitch class consistency streak; and
logic to select an octave-wide band based on analysis of pitches in the frames in the octave selection buffer and to assign the estimated pitch classes of the frames in the octave selection buffer to pitches in the octave-wide band, producing selected pitches.
8. The electronic signal processing component of claim 7 , wherein the streak constructor component and the octave assignment component are implemented using a digital signal processor (DSP).
9. The electronic signal processing component of claim 7 , wherein the streak constructor component and the octave assignment component are implemented using software running on a general purpose central processing unit (hereinafter “CPU”) and the input and output ports are software running on the CPU.
10. The electronic signal processing component of claim 7 , wherein the streak constructor component and the octave assignment component are implemented using a gate array.
11. An electronic signal processing component for pitch determination, including the component of claim 7 , further including:
a pitch adjustment processor coupled between the octave assignment component and the output port and receiving spectrogram data for the data frames, comprising
logic to calculate a first harmonic that would be consistent with harmonic peaks in the spectrogram, to compare the calculated first harmonic to the selected pitch and to adjust the selected pitch if the calculated first harmonic pitch to the selected pitch are within a predetermined adjustment band.
12. The electronic signal processing component for pitch determination, including the component of claim 11 , wherein the pitch adjustment processor further includes:
a peak detection component coupled between the logic to calculate and the output, comprising
logic to search the spectrogram data for the frame to find any additional harmonic peaks in the spectrogram data that are relevant to the adjusted selected pitch, which had been missed in earlier processing, and to make the pitch adjustment processor for further adjustment.
13. An electronic signal processing component for selecting a pitch in frames that represent an audio signal, the component including:
an input port adapted to receive a stream of data frames including at least one pitch estimate per frame;
transformation means, coupled to the input port, for assigning equal pitch classes to pitches in different octaves that have equal positions within their respective octaves;
streak constructor means, coupled to receive the assigned pitch classes from the transformation means, for assigning the frames to one or more pitch class consistency streaks of consecutive frames that have pitch class estimates within a predetermined pitch class margin of one another;
octave assignment means, coupled to the streak constructor means, for selecting a pitch in the frames based on analysis estimated pitch classes in of all or some of the frames in the pitch class consistency streak; and
an output port coupled to the octave assignment means.
14. A volatile or non-volatile computer readable storage medium including program instructions for carrying out a method including:
processing electronically a sequence of frames that include at least one pitch estimate per frame;
transforming the pitch estimates for the frames into pitch class estimates that assign equal pitch classes to pitches in different octaves that have equal positions within their respective octaves;
constructing at least one pitch class consistency streak including pitch classes selected from consecutive frames that have pitch class estimates within a predetermined pitch margin of one another; and
outputting data regarding pitch content of the frames based on at least the pitch classes in the pitch class consistency streak.
15. The volatile or non-volatile computer readable storage medium of claim 14 , wherein at least some of the program instructions are adapted to run on a digital signal processor (hereinafter “DSP”).
16. The volatile or non-volatile computer readable storage medium of claim 15 , wherein the program instructions are adapted to produce a gate array.