Low complexity sub-band speech onset detection (SOD)
Techniques are disclosed for a low-power and low-complexity speech onset detector (SOD) that uses a fractional-band filter structure and spectral subtraction technique to derive sub-band energy profiles to detect the onset of speech in the presence of noise. The SOD derives the sub-band energy profiles by filtering and down-sampling a full-band input audio signal using the fractional-bandwidth filter structure, which may be a low-pass filter with a cut-off frequency that is a fraction of the full bandwidth of the input signal. The SOD flexibly estimates the average noise energy across frames and the current frame speech energy in each sub-band to track noise and speech energy levels across the frames for each of the sub-bands to determine one or more band thresholds used to detect active speech. The sub-band energy profiles leverage any separation in frequency between noise and speech to detect the onset of speech in a target signal.
1 . A method of detecting speech by a device, the method comprising:
receiving an audio signal from one or more microphones;
filtering the audio signal to generate sampled audio data corresponding to a plurality of sampling rates for a plurality of frames of the audio signal;
estimating noise energy and noise-removed energy for a plurality of sub-bands for each of the frames based on the sampled audio data corresponding to the plurality of sampling rates;
tracking noise energy levels and speech energy levels across the plurality of frames based on the estimated noise energy and the estimated noise-removed energy for each of the sub-bands to determine one or more band thresholds for each of the sub-bands based on a maximum of the estimated noise-removed energy; and
determining a presence of speech or noise in each of the frames based on the one or more band thresholds for each of the sub-bands.
2 . The method of claim 1 , wherein filtering the audio signal comprises:
filtering the audio signal to generate a full-band audio signal that is sampled at a full sampling rate;
filtering iteratively the full-band audio signal to generate a hierarchy of one or more successively down-sampled audio signals; and
partitioning the full-band audio signal and the hierarchy of the one or more successively down-sampled audio signals into the plurality of frames to generate the sampled audio data corresponding to the plurality of sampling rates.
3 . The method of claim 2 , wherein filtering iteratively the full-band audio signal to generate the hierarchy of the one or more successively down-sampled audio signals comprises:
down-sampling the full-band audio signal at a fraction of a bandwidth of the full-band audio signal to generate a first down-sampled audio signal belonging to a first level in the hierarchy; and
down-sampling the first down-sampled audio signal belonging to the first level or to a current level in the hierarchy at a fraction of a bandwidth of the first down-sampled audio signal to generate the first down-sampled audio signal belonging to a next lower level in the hierarchy.
4 . The method of claim 1 , wherein estimating noise energy and noise-removed energy for the plurality of sub-bands comprises:
estimating energy levels of the sampled audio data corresponding to the plurality of sampling rates to generate normalized energy levels for a subset of the sub-bands for each of the frames, wherein a sub-band corresponding to a higher sampling rate has a larger bandwidth than a sub-band corresponding to a lower sampling rate; and
determining a difference among the normalized energy levels for the subset of the sub-bands to generate normalized energy levels for a remaining subset of the sub-bands for each of the frames.
5 . The method of claim 4 , wherein estimating the energy levels of the sampled audio data corresponding to the plurality of sampling rates comprises:
aligning the sampled audio data corresponding to the plurality of sampling rates at frames boundaries to produce a frame-aligned sampled audio data; and
determining the normalized energy levels for the subset of the sub-bands corresponding to the plurality of sampling rate for each of the frames based on the frame-aligned sampled audio data.
6 . The method of claim 4 , wherein determining the difference among the normalized energy levels for the subset of the sub-bands comprises:
subtracting the normalized energy level for a smaller sub-band corresponding to a lower sampling rate from the normalized energy level for a larger sub-band corresponding to a higher sampling rate to generate a normalized energy level for a sub-band corresponding to a difference between the smaller sub-band and the larger sub-band;
estimating the noise energy for the plurality of sub-bands for each of the frames based on the normalized energy levels for the plurality of sub-bands; and
subtracting the estimated noise energy from the normalized energy levels for a corresponding one of the plurality of sub-bands to estimate the noise-removed energy for the plurality of sub-bands for each of the frames.
7 . The method of claim 1 , wherein tracking the noise energy levels and the speech energy levels across the plurality of frames comprises:
tracking a mean of the estimated noise energy and the maximum of the estimated noise energy for each of the sub-bands across the plurality of frames;
tracking the maximum of the estimated noise-removed energy for each of the sub-bands across the plurality of frames; and
determining the one or more band thresholds for each of the sub-bands based on the mean of the estimated noise energy, the maximum of the estimated noise energy, and the maximum of the estimated noise-removed energy.
8 . The method of claim 7 , wherein tracking the mean of the estimated noise energy and the maximum of the estimated noise energy comprises:
updating a shadow estimate of the mean of the estimated noise energy or a shadow estimate of the maximum of the estimated noise energy for each of the sub-bands at a plurality of intervals;
determining whether a frame within a frame window contains speech based on the one or more band thresholds for each of the sub-bands at one of the intervals; and
replacing the tracked mean of the estimated noise energy and the maximum of the estimated noise energy for each of the sub-bands with the corresponding shadow estimate when the frame within the frame window is determined to contain speech.
9 . The method of claim 8 , wherein the frame window comprises the plurality of frames and wherein each of the intervals comprises a time interval spanned by the frame window.
10 . The method of claim 1 , wherein determining the presence of speech or noise in each of the frames based on the one or more band thresholds for each of the sub-bands comprises:
determining whether the estimated noise-removed energy in a first frame for one of the sub-bands exceeds a first band threshold;
declaring the first frame as containing noise in response to the estimated noise-removed energy in the first frame for the one sub-band is determined as not exceeding the first band threshold; or
in response to the estimated noise-removed energy in the first frame for the one sub-band is determined as exceeding the first band threshold:
declaring the first frame as containing an onset of speech;
determining whether the estimated noise-removed energy for one of the sub-bands in any frame within the frame window after the first frame exceeds a second band threshold; and
declaring all the frames within the frame window as containing continuing speech when the estimated noise-removed energy for one of the sub-bands in any frame within the frame window exceeds the second band threshold.
11 . An apparatus comprising:
one or more microphones configured to capture sound to produce an audio signal; and
a processing system configured to:
filter the audio signal to generate sampled audio data corresponding to a plurality of sampling rates for a plurality of frames of the audio signal;
estimate noise energy and noise-removed energy for a plurality of sub-bands for each of the frames based on the sampled audio data corresponding to the plurality of sampling rates;
track noise energy levels and speech energy levels across the plurality of frames based on the estimated noise energy and the estimated noise-removed energy for each of the sub-bands to determine one or more band thresholds for each of the sub-bands based on a maximum of the estimated noise-removed energy; and
determine a presence of speech or noise in each of the frames based on the one or more band thresholds for each of the sub-bands.
12 . The apparatus of claim 11 , wherein to filter the audio signal, the processing system is configured to:
filter the audio signal to generate a full-band audio signal that is sampled at a full sampling rate;
filter iteratively the full-band audio signal to generate a hierarchy of one or more successively down-sampled audio signals; and
partition the full-band audio signal and the hierarchy of the down-sampled audio signals into the plurality of frames to generate the sampled audio data corresponding to the plurality of sampling rates.
13 . The apparatus of claim 12 , wherein to filter iteratively the full-band audio signal to generate the hierarchy of successively down-sampled audio signals, the processing system is configured to:
down-sample the full-band audio signal at a fraction of a bandwidth of the full-band audio signal to generate a first down-sampled audio signal belonging to a first level in the hierarchy; and
down-sample the first down-sampled audio signal belonging to the first level or to a current level in the hierarchy at a fraction of a bandwidth of the first down-sampled audio signal to generate the first down-sampled audio signal belonging to a next lower level in the hierarchy.
14 . The apparatus of claim 11 , wherein to estimate the noise energy and the noise-removed energy for the plurality of sub-bands, the processing system is configured to:
estimate energy levels of the sampled audio data corresponding to the plurality of sampling rates to generate normalized energy levels for a subset of the sub-bands for each of the frames, wherein a sub-band corresponding to a higher sampling rate has a larger bandwidth than a sub-band corresponding to a lower sampling rate; and
determine a difference among the normalized energy levels for the subset of the sub-bands to generate normalized energy levels for a remaining subset of the sub-bands for each of the frames.
15 . The apparatus of claim 14 , wherein to estimate the energy levels of the sampled audio data corresponding to the plurality of sampling rates, the processing system is configured:
align the sampled audio data corresponding to the plurality of sampling rates at frames boundaries; and
determine the normalized energy levels for the subset of the sub-bands corresponding to the plurality of sampling rate for each of the frames based on the frame-aligned sampled audio data.
16 . The apparatus of claim 14 , wherein to determine the difference among the normalized energy levels for the subset of the sub-bands, the processing system is configured to:
subtract the normalized energy level for a smaller sub-band corresponding to a lower sampling rate from the normalized energy level for a larger sub-band corresponding to a higher sampling rate to generate a normalized energy level for a sub-band corresponding to a difference between the smaller sub-band and the larger sub-band;
estimate the noise energy for the plurality of sub-bands for each of the frames based on the normalized energy levels for the plurality of sub-bands; and
subtract the estimated noise energy from the normalized energy levels for a corresponding one of the plurality of sub-bands to estimate the noise-removed energy for the plurality of sub-bands for each of the frames.
17 . The apparatus of claim 11 , wherein to track the noise energy levels and the speech energy levels across the plurality of frames, the processing system is configured to:
track a mean of the estimated noise energy and the maximum of the estimated noise energy for each of the sub-bands across the plurality of frames;
track the maximum of the estimated noise-removed energy for each of the sub-bands across the plurality of frames; and
determine the one or more band thresholds for each of the sub-bands based on the mean of the estimated noise energy, the maximum of the estimated noise energy, and the maximum of the estimated noise-removed energy.
18 . The apparatus of claim 17 , wherein to track the mean of the estimated noise energy and the maximum of the estimated noise energy, the processing system is configured to:
update a shadow estimate of the mean of the estimated noise energy or a shadow estimate of the maximum of the estimated noise energy for each of the sub-bands at a plurality of intervals;
determine whether a frame within a frame window contains speech based on the one or more band thresholds for each of the sub-bands at one of the intervals; and
replace the tracked mean of the estimated noise energy and the maximum of the estimated noise energy for each of the sub-bands with the corresponding shadow estimate when the frame within the frame window is determined to contain speech.
19 . The apparatus of claim 18 , wherein the frame window comprises the plurality of frames and wherein each of the intervals comprises a time interval spanned by the frame window.
20 . The apparatus of claim 11 , wherein to determine the presence of speech or noise in each of the frames based on the one or more band thresholds for each of the sub-bands, the processing system is configured to:
determine whether the estimated noise-removed energy in a first frame for one of the sub-bands exceeds a first band threshold;
declare the first frame as containing noise in response to the estimated noise-removed energy in the first frame for the one sub-band is determined as not exceeding the first band threshold; or
in response to the estimated noise-removed energy in the first frame for the one sub-band is determined as exceeding the first band threshold:
declare the first frame as containing an onset of speech;
determine whether the estimated noise-removed energy for one of the sub-bands in any frame within the frame window after the first frame exceeds a second band threshold; and
declare all the frames within the frame window as containing continuing speech when the estimated noise-removed energy for one of the sub-bands in any frame within the frame window exceeds the second band threshold.