Voicing detection modules in a system for automatic transcription of sung or hummed melodies
View Patent ↗The technology disclosed relates to audio signal processing. It includes a series of modules that individually are useful to solve audio signal processing problems. Among the problems addressed are buzz removal, selecting a pitch candidate among pitch candidates based on local continuity of pitch and regional octave consistency, making small adjustments in pitch, ensuring that a selected pitch is consistent with harmonic peaks, determining whether a given frame or region of frames includes harmonic, voiced signal, extracting harmonics from voice signals and detecting vibrato. One environment in which these modules are useful is transcribing singing or humming into a symbolic melody. Another environment that would usefully employ some of these modules is speech processing. Some of the modules, such as buzz removal, are useful in many other environments as well.
1. A method of voicing detection applied to sound that includes both unvoiced and voiced passages, the method including:
processing electronically a sequence of frames that include at least one pitch estimate per frame and one or more magnitude data per frame for harmonics of the pitch estimate;
filtering the sequence of frames for harmonic energy that exceeds a dynamically established harmonic energy threshold, including
calculating a sum per frame that combines at least some of the magnitude data;
dynamically establishing the harmonic energy threshold to be used to identify frames as containing voiced content;
identifying frames as containing voiced content by comparing the calculated sum per frame to the dynamically established harmonic energy threshold;
outputting data regarding voiced content of the sequence of frames based on at least the identification of frames by the filtering for harmonic energy.
2. The method of claim 1 , further including as part of the filtering prior to the calculating, excluding from the sum per frame calculation any of the magnitude data that have a frequency outside of a predetermined frequency band.
3. The method of claim 1 , wherein the sum per frame is calculated using the magnitude squared.
4. The method of claim 1 , further including as part of the filtering after to the calculating, applying a smoother to the calculated sums per frame across the frames.
5. The method of claim 1 , further including adjusting the dynamically established harmonic energy threshold if a rate at which the frames were identified as containing voiced content exceeds a predetermined harmonic energy pass rate, reevaluating the calculated sums per frame against the adjusted dynamically established harmonic energy threshold.
6. The method of claim 1 , wherein dynamically establishing the harmonic energy threshold further includes:
estimating a frequency distribution of occurrences of particular magnitudes across magnitudes present in the magnitude data;
identifying from the frequency distribution a maximum peak with a highest frequency of occurrence and other peaks in frequency of occurrence;
identifying a threshold peak by qualifying the maximum peak of those other peaks that are at least a predetermined height in relation to the maximum peak, choosing the qualifying peak with the greatest magnitude and setting the harmonic energy threshold at the magnitude of the chosen other peak.
7. The method of any of claim 1 , further including:
identifying frames belonging streaks of successive frames with base harmonics present, based on evaluating the magnitude data for one or more of the base harmonics within a predetermined range of the pitch estimate and identifying the streaks of any of the base harmonics present in the successive frames that equal or exceed a predetermined base harmonics streak length; and
outputting data regarding voiced content of the sequence of frames based on at least the identified frames that pass the filtering for harmonic energy and that belong to streaks that exceed the predetermined base harmonics streak length.
8. The method of any of claim 2 , further including:
identifying frames belonging streaks of successive frames with base harmonics present, based on evaluating the magnitude data for one or more of the base harmonics within a predetermined range of the pitch estimate and identifying the streaks of any of the base harmonics present in the successive frames that equal or exceed a predetermined base harmonics streak length; and
outputting data regarding voiced content of the sequence of frames based on at least the identified frames that pass the filtering for harmonic energy and that belong to streaks that exceed the predetermined base harmonics streak length.
9. The method of any of claim 5 , further including:
identifying frames belonging streaks of successive frames with base harmonics present, based on evaluating the magnitude data for one or more of the base harmonics within a predetermined range of the pitch estimate and identifying the streaks of any of the base harmonics present in the successive frames that equal or exceed a predetermined base harmonics streak length; and
outputting data regarding voiced content of the sequence of frames based on at least the identified frames that pass the filtering for harmonic energy and that belong to streaks that exceed the predetermined base harmonics streak length.
10. The method of any of claim 1 , further including:
identifying frames belonging to streaks of successive frames with base harmonics present, based on evaluating the magnitude data for one or more of the base harmonics within a predetermined range of the pitch estimate and identifying the streaks of any of the base harmonics present in the successive frames that equal or exceed a predetermined base harmonics streak length;
identifying frames belonging to streaks of successive frames with extended harmonics present, based on evaluating the magnitude data for a sum of a plurality of the extended harmonics outside the predetermined range of the pitch estimate and identifying the streaks of the extended harmonics present in the successive frames that equal or exceed a predetermined extended harmonics streak length; and
outputting data regarding voiced content of the sequence of frames based on at least the identified frames that pass the filtering for harmonic energy and that have been identified as belonging to a streak of frames containing base or extended harmonic longer than the predetermined base or extended harmonics streak length, respectively.
11. The method of any of claim 2 , further including:
identifying frames belonging to streaks of successive frames with base harmonics present, based on evaluating the magnitude data for one or more of the base harmonics within a predetermined range of the pitch estimate and identifying the streaks of any of the base harmonics present in the successive frames that equal or exceed a predetermined base harmonics streak length;
identifying frames belonging to streaks of successive frames with extended harmonics present, based on evaluating the magnitude data for a sum of a plurality of the extended harmonics outside the predetermined range of the pitch estimate and identifying the streaks of the extended harmonics present in the successive frames that equal or exceed a predetermined extended harmonics streak length; and
outputting data regarding voiced content of the sequence of frames based on at least the identified frames that pass the filtering for harmonic energy and that have been identified as belonging to a streak of frames containing base or extended harmonic longer than the predetermined base or extended harmonics streak length, respectively.
12. The method of any of claim 5 , further including:
identifying frames belonging to streaks of successive frames with base harmonics present, based on evaluating the magnitude data for one or more of the base harmonics within a predetermined range of the pitch estimate and identifying the streaks of any of the base harmonics present in the successive frames that equal or exceed a predetermined base harmonics streak length;
identifying frames belonging to streaks of successive frames with extended harmonics present, based on evaluating the magnitude data for a sum of a plurality of the extended harmonics outside the predetermined range of the pitch estimate and identifying the streaks of the extended harmonics present in the successive frames that equal or exceed a predetermined extended harmonics streak length; and
outputting data regarding voiced content of the sequence of frames based on at least the identified frames that pass the filtering for harmonic energy and that have been identified as belonging to a streak of frames containing base or extended harmonic longer than the predetermined base or extended harmonics streak length, respectively.
13. The method of claim 1 , further including:
evaluating the streaks of successive frames for pitch stability, comprising
determining whether pitch estimates for frames in subsequences of a predetermined sequence length exhibit pitch stability among the frames within a predetermined pitch stability range and retaining those frames exhibiting pitch stability as having voiced content; and
determining whether a single frame with a pitch variance in the subsequence should be forgiven based on pitch stability within predetermined pitch stability range before and within a multiple of the predetermined pitch stability range after the single frame and retaining those frames, including the single frame, as having voiced content;
wherein the outputting data reflects the evaluating of pitch stability.
14. The method of claim 1 , further including filtering out of the identified frames any frames belonging to short streaks of identified frames that are not part of a streak of successive frames that equal or exceed a predetermined identified frame streak length.
15. An electronic signal processing component for detection voicing, the component including:
an input port adapted to receive electronically a stream of data frames including at least one pitch estimate per frame and one or more magnitude data per frame for harmonics of the pitch estimate;
a first filter processor that
calculates a sum per frame that combines at least some of the magnitude data;
dynamically establishes a harmonic energy threshold based on harmonic energy distribution across a plurality of frames;
identifies frames as containing voiced content by comparing the calculated sum per frame to the dynamically established harmonic energy threshold; and
an output port coupled to the filtering processor that outputs data regarding voiced content of the sequence of frames based on at least the identification of frames by the first filtering processor.
16. The component of claim 15 , further including a second filtering processor, operative before the first filtering processor, which excludes from the sum per frame calculation any of the magnitude data that have a frequency outside of a predetermined frequency band.
17. The component of claim 15 , further including a smoother processor operating on the calculated sums per frame, coupled between the first filtering processor and the output port.
18. The component of claim 15 , wherein the first filter processor further includes logic to readjust the dynamically established harmonic energy threshold, the logic adapted to determine if a rate at which the frames were identified as containing voiced content exceeds a predetermined harmonic energy pass rate and to repeat the comparison of the calculated sum per frame against an adjusted dynamically established harmonic energy threshold.
19. The component of claim 15 , wherein the first filtering processor further includes logic to dynamically establish the harmonic energy threshold, the logic adapted to:
estimate a frequency distribution of occurrences of particular magnitudes across magnitudes present in the magnitude data;
identify from the frequency distribution a maximum peak with a highest frequency of occurrence and other peaks in frequency of occurrence;
identify a threshold peak among the other peaks by selecting among the maximum peak and the other peaks that are at least a predetermined height in relation to the maximum peak, choose the peak with the greatest harmonic magnitude and set the harmonic energy threshold at the magnitude of the chosen other peak.
20. The component of claim 15 , further including a streak identification subcomponent, coupled after the first filtering processor, that includes logic to:
identify frames belonging streaks of successive frames with base harmonics present, based on evaluation of the magnitude data for one or more of the base harmonics within a predetermined range of the pitch estimate and identifying the streaks of any of the base harmonics present in the successive frames that equal or exceed a predetermined base harmonics streak length; and
send to the output port data regarding voiced content of the sequence of frames based on at least the identified frames that pass the filtering for harmonic energy and that belong to streaks that exceed the predetermined base harmonics streak length.
21. The component of claim 16 , further including a streak identification subcomponent, coupled after the first filtering processor, that includes logic to:
identify frames belonging to streaks of successive frames with base harmonics present, based on evaluation of the magnitude data for one or more of the base harmonics within a predetermined range of the pitch estimate and identifying the streaks of any of the base harmonics present in the successive frames that equal or exceed a predetermined base harmonics streak length; and
send to the output port data regarding voiced content of the sequence of frames based on at least the identified frames that pass the filtering for harmonic energy and that belong to streaks that exceed the predetermined base harmonics streak length.
22. The component of claim 18 , further including a streak identification subcomponent, coupled after the first filtering processor, that includes logic to:
identify frames belonging to streaks of successive frames with base harmonics present, based on evaluation of the magnitude data for one or more of the base harmonics within a predetermined range of the pitch estimate and identifying the streaks of any of the base harmonics present in the successive frames that equal or exceed a predetermined base harmonics streak length; and
send to the output port data regarding voiced content of the sequence of frames based on at least the identified frames that pass the filtering for harmonic energy and that belong to streaks that exceed the predetermined base harmonics streak length.
23. The component of claim 15 , further including a streak identification subcomponent, coupled after the first filtering processor, that includes logic to:
identify frames belonging to streaks of successive frames with base harmonics present, based on evaluation of the magnitude data for one or more of the base harmonics within a predetermined range of the pitch estimate and identifying the streaks of any of the base harmonics present in the successive frames that equal or exceed a predetermined base harmonics streak length;
identify frames belonging streaks of successive frames with extended harmonics present, based on evaluation of the magnitude data for a plurality of the extended harmonics outside a predetermined range of the pitch estimate and identifying the streaks of any of the extended harmonics present in the successive frames that equal or exceed a predetermined extended harmonics streak length; and
send to the output port data regarding voiced content of the sequence of frames based on at least the identified frames that pass the filtering for harmonic energy and that belong to streaks that exceed the predetermined base harmonics streak length.
24. The component of claim 16 , further including a streak identification subcomponent, coupled after the first filtering processor, that includes logic to:
identify frames belonging streaks of successive frames with base harmonics present, based on evaluation of the magnitude data for one or more of the base harmonics within a predetermined range of the pitch estimate and identifying the streaks of any of the base harmonics present in the successive frames that equal or exceed a predetermined base harmonics streak length;
identify frames belonging streaks of successive frames with extended harmonics present, based on evaluation of the magnitude data for a plurality of the extended harmonics outside a predetermined range of the pitch estimate and identifying the streaks of any of the extended harmonics present in the successive frames that equal or exceed a predetermined extended harmonics streak length; and
send to the output port data regarding voiced content of the sequence of frames based on at least the identified frames that pass the filtering for harmonic energy and that belong to streaks that exceed the predetermined base harmonics streak length.
25. The component of claim 18 , further including a streak identification subcomponent, coupled after the first filtering processor, that includes logic to:
identify frames belonging streaks of successive frames with base harmonics present, based on evaluation of the magnitude data for one or more of the base harmonics within a predetermined range of the pitch estimate and identifying the streaks of any of the base harmonics present in the successive frames that equal or exceed a predetermined base harmonics streak length;
identify frames belonging streaks of successive frames with extended harmonics present, based on evaluation of the magnitude data for a plurality of the extended harmonics outside a predetermined range of the pitch estimate and identifying the streaks of any of the extended harmonics present in the successive frames that equal or exceed a predetermined extended harmonics streak length; and
send to the output port data regarding voiced content of the sequence of frames based on at least the identified frames that pass the filtering for harmonic energy and that belong to streaks that exceed the predetermined base harmonics streak length.
26. An signal processing component for detection voicing, the component including:
an input port adapted to receive a stream of data frames including at least one pitch estimate per frame;
first filtering means for identifying frames as containing voiced content by comparing a calculated sum per frame to a dynamically established harmonic energy threshold; and
an output port coupled to the first filtering means that outputs data regarding voiced content of the sequence of frames based on at least the identification of frames by the first filtering means.
27. The signal processing component of claim 26 , further including:
base harmonic presence means, coupled to the first filtering means and the output port, for identifying frames belonging streaks of successive frames with base harmonics present, based on evaluating the magnitude data for one or more of the base harmonics within a predetermined range of the pitch estimate and identifying the streaks of any of the base harmonics present in the successive frames that equal or exceed a predetermined base harmonics streak length.
28. The signal processing component of claim 27 , further including:
extended harmonic presence means, coupled to the first filtering means and the output port and cooperating with the base harmonic presence means, for identifying frames belonging streaks of successive frames with extended harmonics present, based on evaluating the magnitude data for a plurality of the extended harmonics outside a predetermined range of the pitch estimate and identifying the streaks of any of the base harmonics present in the successive frames that equal or exceed a predetermined base harmonics streak length.
29. The signal processing component of claim 26 , further including:
pitch stability means, coupled before the output port, for evaluating the streaks of successive frames for pitch stability and passing the results of the evaluating to the output port.
30. The signal processing component of claim 26 , further including:
short streak exclusion means, coupled before the output port, for filtering out of the identified frames any frames belonging to short streaks of identified frames that are not part of a streak of successive frames that equal or exceed a predetermined identified frame streak length.
31. A volatile or non-volatile computer readable storage medium including program instructions for carrying out a method including:
processing electronically a sequence of frames that include at least one pitch estimate per frame and one or more magnitude data per frame for harmonics of the pitch estimate;
filtering the sequence of frames for harmonic energy that exceeds a dynamically established harmonic energy threshold, including
calculating a sum per frame that combines at least some of the magnitude data;
dynamically establishing the harmonic energy threshold to be used to identify frames as containing voiced content;
identifying frames as containing voiced content by comparing the calculated sum per frame to the dynamically established harmonic energy threshold;
outputting data regarding voiced content of the sequence of frames based on at least the identification of frames by the filtering for harmonic energy.
32. The computer readable storage medium of claim 31 , wherein at least some of the program instructions are adapted to run on a digital signal processor (hereinafter “DSP”).
33. The computer readable storage medium of claim 31 , wherein the program instructions are adapted to produce a gate array.