Granted Patent B2

US 12,293,770 · App. 17/685,042 · Granted May 6, 2025

Inventors: Rui Zhu (Shenzhen, CN); Juan Juan Li (Shenzhen, CN); Yan Nan Wang (Shenzhen, CN); Yue Peng Li (Shenzhen, CN)

Assignee: TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED

G10L21/0232G10L25/12G10L25/18G10L25/21G10L25/30G10L2021/02082

Abstract

Claims (89)

1. A speech signal dereverberation processing method, executed by at least one processor, the method comprising:

extracting an amplitude spectrum feature and a phase spectrum feature of a current frame in an original speech signal;

extracting subband amplitude spectrums from the amplitude spectrum feature corresponding to the current frame;

determining, based on the subband amplitude spectrums and a reverberation strength distribution associated with the current frame and by using a first model, a reverberation strength indicator corresponding to the current frame, the first model being a first neural network model that is trained using reverberated band amplitude spectrum, clean speech band amplitude spectrum, and a reverberation-to-clean-speech energy ratio, with the reverberation-to-clean-speech energy ratio used as a training target;

determining, based on the subband amplitude spectrums and the reverberation strength indicator, and by using a second model, a clean speech subband spectrum corresponding to the current frame, wherein the second model is a regressive reverberation strength prediction algorithm model based on a history frame; and

obtaining a dereverberated clean speech signal by performing signal conversion on the clean speech subband spectrum and the phase spectrum feature corresponding to the current frame.

2. The method of claim 1 , wherein the determining the reverberation strength indicator corresponding to the current frame comprises:

predicting, by using the first model, a clean speech energy ratio corresponding to the subband amplitude spectrums; and

determining, based on the clean speech energy ratio and the reverberation strength distribution associated with the current frame, the reverberation strength indicator corresponding to the current frame.

3. The method of claim 2 , wherein the predicting the clean speech energy ratio corresponding to the subband amplitude spectrums comprises:

extracting a dimension feature of the subband amplitude spectrums by using an input layer of the first model;

extracting representation information of the subband amplitude spectrums based on the dimension feature and by using a prediction layer of the first model; and

determining the clean speech energy ratio of the subband amplitude spectrums based on the representation information; and

wherein the determining the reverberation strength indicator corresponding to the current frame comprises:

outputting, by using an output layer of the first model and based on the clean speech energy ratio corresponding to the subband amplitude spectrums, the reverberation strength indicator corresponding to the current frame.

4. The method of claim 1 , wherein the determining the clean speech subband spectrum corresponding to the current frame comprises:

determining a posterior signal-to-interference ratio of the current frame based on the amplitude spectrum feature of the current frame and by using the second model;

determining a prior signal-to-interference ratio of the current frame based on the posterior signal-to-interference ratio and the reverberation strength indicator; and

obtaining a clean speech subband amplitude spectrum corresponding to the current frame by performing filtering enhancement processing on the subband amplitude spectrums of the current frame based on the prior signal-to-interference ratio.

5. The method of claim 4 , wherein the determining the posterior signal-to-interference ratio of the current frame comprises:

extracting a steady noise amplitude spectrum corresponding to each subband in the current frame by using the second model;

extracting a steady reverberation amplitude spectrum corresponding to each subband in the current frame by using the second model; and

determining the posterior signal-to-interference ratio of the current frame based on the steady noise amplitude spectrum, the steady reverberation amplitude spectrum, and the subband amplitude spectrums.

6. The method of claim 5 , wherein the determining the posterior signal-to-interference ratio of the current frame comprises:

obtaining a clean speech amplitude spectrum of a previous frame; and

estimating the posterior signal-to-interference ratio of the current frame based on the clean speech amplitude spectrum of the previous frame and based on the steady noise amplitude spectrum, the steady reverberation amplitude spectrum, and the subband amplitude spectrums.

7. The method of claim 1 , wherein the extracting the amplitude spectrum feature and the phase spectrum feature corresponding to the current frame in the original speech signal comprises:

obtaining the amplitude spectrum feature and the phase spectrum feature corresponding to the current frame in the original speech signal by performing framing and windowing processing on the original speech signal; and

wherein the extracting subband amplitude spectrums from the amplitude spectrum feature corresponding to the current frame comprises:

obtaining a preset band coefficient; and

obtaining the subband amplitude spectrums corresponding to the current frame by performing band division on the amplitude spectrum feature of the current frame based on a band coefficient.

8. The method of claim 1 , wherein the obtaining the dereverberated clean speech signal comprises:

obtaining a clean speech amplitude spectrum corresponding to the current frame by performing inverse constant transform on the clean speech subband spectrum according to a preset band coefficient; and

obtaining the dereverberated clean speech signal by performing time-to-frequency conversion on the clean speech amplitude spectrum and the phase spectrum feature corresponding to the current frame, to obtain the dereverberated clean speech signal.

9. The method of claim 1 , wherein the first model is trained by:

obtaining reverberated speech data and clean speech data corresponding to the reverberated speech data, and generating training sample data by using the reverberated speech data and the clean speech data;

determining the reverberation-to-clean-speech energy ratio of the reverberated speech data to the clean speech data as the training target;

extracting the reverberated band amplitude spectrum corresponding to the reverberated speech data, and extracting the clean speech band amplitude spectrum of the clean speech data; and

training the first model by using the reverberated band amplitude spectrum, the clean speech band amplitude spectrum, and the training target.

10. The method of claim 9 , wherein the training the first model by using the reverberated band amplitude spectrum, the clean speech band amplitude spectrum, and the training target comprises:

obtaining a training result by inputting the reverberated band amplitude spectrum and the clean speech band amplitude spectrum to a preset network model; and

obtaining a required first model by adjusting a parameter of a preset neural network model based on a difference between the training result and the training target, and continuing the training, until a training condition is met.

11. A speech signal dereverberation processing apparatus, comprising:

at least one memory configured to store computer program code; and

at least one processor configured to access said computer program code and operate as instructed by said computer program code, said computer program code comprising:

first extracting code configured to cause the at least one processor to extract an amplitude spectrum feature and a phase spectrum feature of a current frame in an original speech signal;

second extracting code configured to cause the at least one processor to extract subband amplitude spectrums from the amplitude spectrum feature corresponding to the current frame;

first determining code configured to cause the at least one processor to determine, based on the subband amplitude spectrums and a reverberation strength distribution associated with the current frame and by using a first model, a reverberation strength indicator corresponding to the current frame, the first model being a first neural network model that is trained using reverberated band amplitude spectrum, clean speech band amplitude spectrum, and a reverberation-to-clean-speech energy ratio, with the reverberation-to-clean-speech energy ratio used as a training target;

second determining code configured to cause the at least one processor to determine, based on the subband amplitude spectrums and the reverberation strength indicator, and by using a second model, a clean speech subband spectrum corresponding to the current frame, wherein the second model is a regressive reverberation strength prediction algorithm model based on a history frame; and

obtaining code configured to cause the at least one processor to obtain a dereverberated clean speech signal by performing signal conversion on the clean speech subband spectrum and the phase spectrum feature corresponding to the current frame.

12. The apparatus of claim 11 , wherein the first determining code is further configured to cause the at least one processor to:

predict, using the first model, a clean speech energy ratio corresponding to the subband amplitude spectrums; and

determine, based on the clean speech energy ratio and the reverberation strength distribution associated with the current frame, the reverberation strength indicator corresponding to the current frame.

13. The apparatus of claim 12 , wherein the first determining code is further configured to cause the at least one processor to predict the clean speech energy ratio corresponding to the subband amplitude spectrums by:

extracting a dimension feature of the subband amplitude spectrums by using an input layer of the first model;

extracting representation information of the subband amplitude spectrums based on the dimension feature and by using a prediction layer of the first model; and

determining the clean speech energy ratio of the subband amplitude spectrums based on the representation information; and

wherein the first determining code is further configured to cause the at least one processor to:

output, using an output layer of the first model and based on the clean speech energy ratio corresponding to the subband amplitude spectrums, the reverberation strength indicator corresponding to the current frame.

14. The apparatus of claim 11 , wherein the second determining code is further configured to cause the at least one processor to:

determine a posterior signal-to-interference ratio of the current frame based on the amplitude spectrum feature of the current frame and by using the second model;

determine a prior signal-to-interference ratio of the current frame based on the posterior signal-to-interference ratio and the reverberation strength indicator; and

obtain a clean speech subband amplitude spectrum corresponding to the current frame by performing filtering enhancement processing on the subband amplitude spectrums of the current frame based on the prior signal-to-interference ratio.

15. The apparatus of claim 14 , wherein the second determining code is further configured to cause the at least one processor to determine the posterior signal-to-interference ratio of the current frame by:

extracting a steady noise amplitude spectrum corresponding to each subband in the current frame by using the second model;

extracting a steady reverberation amplitude spectrum corresponding to each subband in the current frame by using the second model; and

determining the posterior signal-to-interference ratio of the current frame based on the steady noise amplitude spectrum, the steady reverberation amplitude spectrum, and the subband amplitude spectrums.

16. The apparatus of claim 15 , wherein the second determining code is further configured to cause the at least one processor to determine the posterior signal-to-interference ratio of the current frame by:

obtaining a clean speech amplitude spectrum of a previous frame; and

estimating the posterior signal-to-interference ratio of the current frame based on the clean speech amplitude spectrum of the previous frame and based on the steady noise amplitude spectrum, the steady reverberation amplitude spectrum, and the subband amplitude spectrums.

17. The apparatus of claim 11 , wherein the first extracting code is further configured to cause the at least one processor to:

obtain the amplitude spectrum feature and the phase spectrum feature corresponding to the current frame in the original speech signal by performing framing and windowing processing on the original speech signal; and

wherein the second extracting code is further configured to cause the at least one processor to:

obtain a preset band coefficient; and

obtain the subband amplitude spectrums corresponding to the current frame by performing band division on the amplitude spectrum feature of the current frame based on a band coefficient.

18. The apparatus of claim 11 , wherein the obtaining code is further configured to cause the at least one processor to:

obtain a clean speech amplitude spectrum corresponding to the current frame by performing inverse constant transform on the clean speech subband spectrum according to a preset band coefficient; and

obtain the dereverberated clean speech signal by performing time-to-frequency conversion on the clean speech amplitude spectrum and the phase spectrum feature corresponding to the current frame, to obtain the dereverberated clean speech signal.

19. The apparatus of claim 11 , wherein the first model is trained by:

obtaining reverberated speech data and clean speech data, and generating training sample data by using the reverberated speech data and the clean speech data;

determining the reverberation-to-clean-speech energy ratio of the reverberated speech data to the clean speech data as the training target;

extracting the reverberated band amplitude spectrum corresponding to the reverberated speech data, and extracting the clean speech band amplitude spectrum of the clean speech data; and

training the first model by using the reverberated band amplitude spectrum, the clean speech band amplitude spectrum, and the training target.

20. A non-transitory computer-readable storage medium storing computer instructions that, when executed by at least one processor of a speech signal dereverberation processing device, cause the at least one processor to:

extract an amplitude spectrum feature and a phase spectrum feature of a current frame in an original speech signal;

extract subband amplitude spectrums from the amplitude spectrum feature corresponding to the current frame;

determine, based on the subband amplitude spectrums and a reverberation strength distribution associated with the current frame and by using a first model, a reverberation strength indicator corresponding to the current frame, wherein the first model is a first neural network model that is trained using reverberated band amplitude spectrum, clean speech band amplitude spectrum, and a reverberation-to-clean-speech energy ratio, with the reverberation-to-clean-speech energy ratio used as a training target;

determine, based on the subband amplitude spectrums and the reverberation strength indicator, and by using a second model, a clean speech subband spectrum corresponding to the current frame, wherein the second model is a regressive reverberation strength prediction algorithm model based on a history frame; and

obtain a dereverberated clean speech signal by performing signal conversion on the clean speech subband spectrum and the phase spectrum feature corresponding to the current frame.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (20)