Automatic leveling of speech content
Embodiments are disclosed for automatic leveling of speech content. In an embodiment, a method comprises: receiving, using one or more processors, frames of an audio recording including speech and non-speech content; for each frame: determining, using the one or more processors, a speech probability; analyzing, using the one or more processors, a perceptual loudness of the frame; obtaining, using the one or more processors, a target loudness range for the frame; computing, using the one or more processors, gains to apply to the frame based on the target loudness range and the perceptual loudness analysis, where the gains include dynamic gains that change frame-by-frame and that are scaled based on the speech probability; and applying the gains to the frame so that a resulting loudness range of the speech content in the audio recording fits within the target loudness range.
1. A method comprising:
receiving, using one or more processors, frames of an audio recording including speech and non-speech, where the speech includes speech from multiple speakers with different signal-to-noise ratios (SNRs);
detecting speech frames in the speech content and their respective speech probabilities;
segmenting the speech frames into separate streams according to the identities of the multiple speakers, wherein segmenting is based on time indices to indicate where each of the multiple speakers is actively dominant in the audio recording;
for each frame in each one of the separate streams:
analyzing, using the one or more processors, a perceptual loudness of the frame, the analyzing including determining an integrated loudness and a momentary loudness;
obtaining, using the one or more processors, a target loudness range for the frame;
in response to a first condition being met,
setting a dynamic gain for the frame as a function of the integrated loudness, the target loudness range, the momentary loudness and a non-linear function of the speech probability for the frame, wherein the non-linear function of the speech probability is configured to avoid boosting non-speech parts of the audio;
in response to a second condition being met, setting the dynamic gain for the frame as a function of the integrated loudness, the target loudness range, and the momentary loudness for the frame, wherein the second condition is different than the first condition;
and otherwise, when both the first and second conditions are not met, setting the dynamic range to zero; and
applying, using the one or more processors, the dynamic gains to the frame so that a resulting loudness range of the speech content in the audio recording fits within the target loudness range.
2. The method of claim 1 , further comprising:
computing, using the one or more processors, a static gain that is applied to all the frames.
3. The method of claim 2 , where the static gain is the difference between an integrated loudness and a target loudness.
4. The method of claim 2 , where the gain applied to each frame is the sum of the static gain and the dynamic gain for the frame.
5. The method of claim 1 , wherein the dynamic gains are computed as a continuous function of a distance between the perceptual loudness of each frame and an integrated loudness.
6. The method of claim 1 , wherein the dynamic gains of the frames within a desired loudness range with respect to the integrated loudness are unity, and the dynamic gains applied to frames outside the desired loudness range are computed as the difference between the frame's loudness value and the nearest boundary of the desired loudness range.
7. The method of claim 1 , where the dynamic gains are multiplied by a coefficient between 0.0 and 1.0.
8. The method of claim 1 , where the speech probability is computed by a neural network.
9. The method of claim 1 , where the speech probability is a function of a broadband energy level of each frame.
10. The method of claim 1 , further comprising:
estimating a signal-to-noise ratio (SNR); and
modifying the speech probability based at least in part on the estimated SNR.
11. The method of claim 10 , where the speech probability is determined by a voice activity detector (VAD), and the method further comprises:
adjusting a sensitivity of the VAD to increase discrimination between speech and non-speech when the estimated SNR indicates the speech content is clean.
12. The method of claim 1 , further comprising:
estimating a signal-to-noise ratio (SNR); and
adjusting the target loudness based on the estimated SNR so that a small dynamic range is only achieved when the speech content is clean.
13. The method of claim 10 , where the dynamic gains are multiplied by a coefficient between 0 and 1, and the coefficient is a function of the SNR.
14. The method of claim 1 , where the speech probability can be modified through a sigmoid function, and wherein a parameter of the sigmoid function is either manually fixed or automatically adapted based on the estimated SNR of the speech content.
15. The method of claim 1 , where the speech probability is a function of the energy level of each frame in a specific frequency band.
16. The method of claim 1 , where the perceptual loudness of each frame is computed at recording time and stored.
17. The method of claim 1 , where the speech probability is computed at recording time and stored.
18. The method of claim 12 , wherein the estimated SNR is used to control a parameter of the non-linear function of the speech probability.
19. A system comprising:
one or more processors; and
a non-transitory computer-readable medium storing instructions that, upon execution by the one or more processors, cause the one or more processors to perform operations of the method of claim 1 .
20. A non-transitory, computer-readable medium storing instructions that, upon execution by one or more processors, cause the one or more processors to perform operations of the method of claim 1 .