Whispering voice recovery method, apparatus and device, and readable storage medium
A method, an apparatus and a device for converting a whispered speech, and a readable storage medium are provided. The method is implemented based on the whispered speech converting model. The whispered speech converting model is trained in advance by using recognition results and whispered speech training acoustic features of whispered speech training data as samples and using normal speech acoustic features of normal speech data parallel to the whispered speech training data as sample labels. A whispered speech acoustic feature and a preliminary recognition result of whispered speech data are acquired, then the whispered speech acoustic feature and the preliminary recognition result are inputted into a preset whispered speech converting model to acquire a normal speech acoustic feature outputted by the model. In this way, the whispered speech can be converted to a normal speech.
1. A method for converting a whispered speech, comprising:
acquiring a whispered speech acoustic feature of whispered speech data, and acquiring a preliminary recognition result of the whispered speech data, wherein the preliminary recognition result is a recognition result for the whispered speech data; and
inputting the whispered speech acoustic feature and the preliminary recognition result into a preset whispered speech converting model to acquire a normal speech acoustic feature outputted by the whispered speech converting model, wherein
the whispered speech converting model is trained in advance by using recognition results of whispered speech training data and whispered speech training acoustic features of the whispered speech training data as samples and using normal speech acoustic features of normal speech data parallel to the whispered speech training data as sample labels,
wherein the acquiring the preliminary recognition result of the whispered speech data comprises:
inputting the whispered speech acoustic feature into a preset whispered speech recognition model to acquire a whispered speech recognition result outputted by the whispered speech recognition model as the preliminary recognition result of the whispered speech data, wherein the whispered speech recognition model is acquired by taking a normal speech recognition model as an initial model and training the initial model by using whispered speech training acoustic features labeled with recognition results of whispered speech training data.
2. The method according to claim 1 , further comprising:
determining a final recognition result of the whispered speech data based on the normal speech acoustic feature.
3. The method according to claim 1 , further comprising:
acquiring lip shape image data matched with the whispered speech data, wherein
the acquiring the preliminary recognition result of the whispered speech data further comprises:
inputting the lip shape image data into a preset lip shape recognition model to acquire a lip shape recognition result outputted by the lip shape recognition model, wherein the lip shape recognition model is trained in advance by using lip shape image training data labeled with lip shape recognition results; and
merging the whispered speech recognition result and the lip shape recognition result to acquire a merged recognition result as the preliminary recognition result of the whispered speech data.
4. The method according to claim 3 , further comprising:
performing lip detection on each frame of lip shape images of the lip shape image data to acquire a lip region; and
extracting the lip region from the lip shape image comprising the lip region, performing an image regularization process on the lip region to acquire regularized lip shape image data as an input of the lip shape recognition model.
5. The method according to claim 1 , wherein the acquiring the whispered speech acoustic feature of the whispered speech data comprises:
segmenting the whispered speech data into frames to acquire a plurality of frames of whispered speech data;
performing a pre-emphasis process on each frame of whispered speech data to acquire a frame of pre-emphasis processed whispered speech data; and
extracting a spectrum feature of each frame of pre-emphasis processed whispered speech data, wherein the spectrum feature comprises one or more of a LogFilter Bank Energy feature, a Mel Frequency Cepstrum Coefficient feature, and a Perceptual Linear Predictive feature.
6. The method according to claim 1 , wherein the inputting the whispered speech acoustic feature and the preliminary recognition result into the preset whispered speech converting model to acquire the normal speech acoustic feature outputted by the whispered speech converting model comprises:
inputting the whispered speech acoustic feature and the preliminary recognition result into a whispered speech converting model having a recurrent neural network type, to acquire the normal speech acoustic feature outputted by the whispered speech converting model.
7. The method according to claim 1 , wherein the inputting the whispered speech acoustic feature and the preliminary recognition result into the preset whispered speech converting model to acquire the normal speech acoustic feature outputted by the whispered speech converting model comprises:
inputting the whispered speech acoustic feature and the preliminary recognition result into a whispered speech converting model having a codec type of attention mechanism;
encoding the whispered speech acoustic feature and the preliminary recognition result by using a coding layer of the whispered speech converting model, to acquire an encoded whispered speech acoustic feature and an encoded preliminary recognition result;
performing coefficient linear weighting on the encoded whispered speech acoustic feature by using an attention layer of the whispered speech converting model, to acquire a weighted whispered speech acoustic feature at a current time instant; and
taking the encoded preliminary recognition result, the weighted whispered speech acoustic feature at the current time instant, and an output of a decoding layer of the whispered speech converting model at the previous time instant as an input of the decoding layer at the current time instant, to acquire an output of the decoding layer at the current time instant as the normal speech acoustic feature.
8. The method according to claim 2 , wherein the determining the final recognition result of the whispered speech data based on the normal speech acoustic feature comprises:
inputting the normal speech acoustic feature into a preset normal speech recognition model to acquire a normal speech recognition result outputted by the normal speech recognition model; and
determining the normal speech recognition result as the final recognition result of the whispered speech data.
9. The method according to claim 2 , wherein the determining the final recognition result of the whispered speech data based on the normal speech acoustic feature comprises:
inputting the normal speech acoustic feature into a preset normal speech recognition model to acquire a normal speech recognition result outputted by the normal speech recognition model;
determining whether a preset iteration termination condition is met;
determining the normal speech recognition result as the final recognition result of the whispered speech data, in a case that the preset iteration termination condition is met; and
determining the normal speech recognition result as the preliminary recognition result and returning to perform the process of inputting the whispered speech acoustic feature and the preliminary recognition result into the preset whispered speech converting model, in a case that the preset iteration termination condition is not met.
10. An apparatus for converting a whispered speech, comprising:
a whispered speech acoustic feature acquiring unit, configured to acquire a whispered speech acoustic feature of whispered speech data;
a preliminary recognition result acquiring unit, configured to acquire a preliminary recognition result of the whispered speech data, wherein the preliminary recognition result is a recognition result for the whispered speech data;
a whispered speech converting unit, configured to input the whispered speech acoustic feature and the preliminary recognition result into a preset whispered speech converting model to acquire a normal speech acoustic feature outputted by the whispered speech converting model, wherein
the whispered speech converting model is trained in advance by using recognition results of whispered speech training data and whispered speech training acoustic features of the whispered speech training data as samples and using normal speech acoustic features of normal speech data parallel to the whispered speech training data as sample labels,
wherein the preliminary recognition result acquiring unit comprises:
a first preliminary recognition result acquiring subunit, configured to input the whispered speech acoustic feature into a preset whispered speech recognition model to acquire a whispered speech recognition result outputted by the whispered speech recognition model as the preliminary recognition result of the whispered speech data, wherein the whispered speech recognition model is acquired by taking a normal speech recognition model as an initial model and training the initial model by using whispered speech training acoustic features labeled with recognition results of whispered speech training data.
11. The apparatus according to claim 10 , further comprising:
a final recognition result determining unit, configured to determine a final recognition result of the whispered speech data based on the normal speech acoustic feature.
12. The apparatus according to claim 10 , further comprising:
a lip shape image data acquiring unit, configured to acquire lip shape image data matched with the whispered speech data;
the preliminary recognition result acquiring unit further comprises:
a second preliminary recognition result acquiring subunit, configured to input the lip shape image data into a preset lip shape recognition model to acquire a lip shape recognition result outputted by the lip shape recognition model, wherein the lip shape recognition model is trained in advance by using lip shape image training data labeled with lip shape recognition results; and
a third preliminary recognition result acquiring subunit, configured to merge the whispered speech recognition result and the lip shape recognition result to acquire a merged recognition result as the preliminary recognition result of the whispered speech data.
13. The apparatus according to claim 12 , further comprising:
a lip detecting unit, configured to perform lip detection on each frame of lip shape images of the lip shape image data to acquire a lip region; and
an image processing unit, configured to extract the lip region from the lip shape image comprising the lip region and perform an image regularization process on the lip region to acquire regularized lip shape image data as an input of the lip shape recognition model.
14. The apparatus according to claim 10 , wherein the whispered speech acoustic feature acquiring unit comprises:
a framing processing unit, configured to segment the whispered speech data into frames to acquire a plurality of frames of whispered speech data;
a pre-emphasis processing unit, configured to perform a pre-emphasis process on each frame of whispered speech data to acquire a frame of pre-emphasis processed whispered speech data; and
a spectrum feature extracting unit, configured to extract a spectrum feature of each frame of pre-emphasis processed whispered speech data, wherein the spectrum feature comprises one or more of a LogFilter Bank Energy feature, a Mel Frequency Cepstrum Coefficient feature, and a Perceptual Linear Predictive feature.
15. The apparatus according to claim 10 , wherein the whispered speech converting unit comprises:
a recursive processing unit, configured to input the whispered speech acoustic feature and the preliminary recognition result into a whispered speech converting model having a recurrent neural network type, to acquire the normal speech acoustic feature outputted by the whispered speech converting model.
16. The apparatus according to claim 10 , wherein the whispered speech converting unit comprises a codec processing unit, the codec processing unit comprises:
a first codec processing subunit, configured to input the whispered speech acoustic feature and the preliminary recognition result into a whispered speech converting model having a codec type of attention mechanism;
a second codec processing subunit, configured to encode the whispered speech acoustic feature and the preliminary recognition result by using a coding layer of the whispered speech converting model, to acquire an encoded whispered speech acoustic feature and an encoded preliminary recognition result;
a third codec processing subunit, configured to perform coefficient linear weighting on the encoded whispered speech acoustic feature by using an attention layer of the whispered speech converting model, to acquire a weighted whispered speech acoustic feature at a current time instant; and
a fourth codec processing subunit, configured to take the encoded preliminary recognition result, the weighted whispered speech acoustic feature at the current time instant, and an output of a decoding layer of the whispered speech converting model at the previous time instant as an input of the decoding layer at the current time instant, to acquire an output of the decoding layer at the current time instant as the normal speech acoustic feature.
17. The apparatus according to claim 11 , wherein the final recognition result determining unit comprises:
a normal speech recognition unit, configured to input the normal speech acoustic feature into a preset normal speech recognition model to acquire a normal speech recognition result outputted by the normal speech recognition model; and
a first result determining unit, configured to determine the normal speech recognition result as the final recognition result of the whispered speech data.
18. The apparatus according to claim 11 , wherein the final recognition result determining unit comprises:
a normal speech recognition unit, configured to input the normal speech acoustic feature into a preset normal speech recognition model to acquire a normal speech recognition result outputted by the normal speech recognition model;
an iteration determining unit, configured to determine whether a preset iteration termination condition is met;
a second result determining unit, configured to determine the normal speech recognition result as the final recognition result of the whispered speech data, in a case that the iteration determining unit determines that the preset iteration termination condition is met; and
a third result determining unit, configured to determine the normal speech recognition result as the preliminary recognition result and return to perform the process of inputting the whispered speech acoustic feature and the preliminary recognition result into the preset whispered speech converting model, in a case that the iteration determining unit determines that the preset iteration termination condition is not met.