Voice conversion system and training method therefor
View Patent ↗The present disclosure proposes a speech conversion scheme for non-parallel corpus training, to get rid of dependence on parallel text and resolve a technical problem that it is difficult to achieve speech conversion under conditions that resources and equipment are limited. A voice conversion system and a training method therefor are included. Compared with the prior art, according to the embodiments of the present disclosure: a trained speaker-independent automatic speech recognition model can be used for any source speaker, that is, the speaker is independent; and bottleneck features of audio are more abstract as compared with phonetic posteriorGram features, can reflect decoupling of spoken content and timbre of the speaker, and meanwhile are not closely bound with a phoneme class, and are not in a clear one-to-one correspondence relationship. In this way, a problem of inaccurate pronunciation caused by a recognition error in ASR is relieved to some extent. Pronunciation accuracy of audio obtained by performing voice conversion by the bottleneck feature is obviously higher than that of a phonetic posteriorGram based method, and timbre is not significantly different. By means of a transfer learning mode, dependence on training corpus can be greatly reduced.
1. A voice conversion system, comprising:
a speaker-independent automatic speech recognition model, comprising at least a bottleneck layer, configured to: convert a mel-scale frequency cepstral coefficients feature of an inputted source speech into a bottleneck feature of the source speech through the bottleneck layer, and output the bottleneck feature of the source speech to an Attention voice conversion network through the bottleneck layer;
where a training method for the speaker-independent automatic speech recognition model comprises:
inputting a number of a character encoding to which a word in a multi-speaker speech recognition training corpus is converted, together with a mel-scale frequency cepstral coefficients feature of the multi-speaker speech recognition training corpus, to the speaker-independent automatic speech recognition model; executing a backward propagation algorithm; and performing iterative optimization until the speaker-independent automatic speech recognition model is converged;
the Attention voice conversion network configured to convert the bottleneck feature of the source speech into a mel-scale frequency cepstral coefficients feature in conformity with a target speech; and
a neural network vocoder configured to convert the mel-scale frequency cepstral coefficients feature in conformity with the target speech into and output audio.
2. A training method for a voice conversion system, applicable to the voice conversion system according to claim 1 , wherein a training method for the speaker-independent automatic speech recognition model comprises:
inputting a number of a character encoding to which a word in a multi-speaker speech recognition training corpus is converted, together with a mel-scale frequency cepstral coefficients feature of the multi-speaker speech recognition training corpus, to the speaker-independent automatic speech recognition model;
executing a backward propagation algorithm; and
performing iterative optimization until the speaker-independent automatic speech recognition model is converged.
3. The training method for the voice conversion system according to claim 2 , wherein the training method for the speaker-independent automatic speech recognition model further comprises preprocessing the multi-speaker speech recognition training corpus, wherein the preprocessing comprises de-blanking and normalizing.
4. The training method for the voice conversion system according to claim 3 , further comprising extracting a parameter to obtain the mel-scale frequency cepstral coefficients feature of the multi-speaker speech recognition training corpus, a mel-scale frequency cepstral coefficients feature of a target speech, and a mel-scale frequency cepstral coefficients feature of a source speech.
5. The training method for the voice conversion system according to claim 2 , characterized by further comprising extracting a parameter to obtain the mel-scale frequency cepstral coefficients feature of the multi-speaker speech recognition training corpus, a mel-scale frequency cepstral coefficients feature of a target speech, and a mel-scale frequency cepstral coefficients feature of a source speech.
6. A training method for a voice conversion system, applicable to the voice conversion system according to claim 1 , wherein a training method for the Attention voice conversion network comprises:
converting a mel-scale frequency cepstral coefficients feature of a target speech into a bottleneck feature of the target speech;
inputting the bottleneck feature of the target speech to a basic Attention voice conversion network, with the mel-scale frequency cepstral coefficients feature corresponding to the target speaker as a ground truth, and
training the Attention voice conversion network by a method of Deep Transfer Learning.
7. The training method for the voice conversion system according to claim 6 , wherein the converting of the mel-scale frequency cepstral coefficients feature of the target speech into the bottleneck feature of the target speech is performed by using the speaker-independent automatic speech recognition model that is pre-trained.
8. The training method for the voice conversion system according to claim 7 , further comprising preprocessing the target speech, wherein the preprocessing comprises de-blanking and normalizing.
9. The training method for the voice conversion system according to claim 7 , further comprising extracting a parameter to obtain the mel-scale frequency cepstral coefficients feature of the multi-speaker speech recognition training corpus, a mel-scale frequency cepstral coefficients feature of a target speech, and a mel-scale frequency cepstral coefficients feature of a source speech.
10. The training method for the voice conversion system according to claim 6 , further comprising preprocessing the target speech, wherein the preprocessing comprises de-blanking and normalizing.
11. The training method for the voice conversion system according to claim 6 , further comprising extracting a parameter to obtain the mel-scale frequency cepstral coefficients feature of the multi-speaker speech recognition training corpus, a mel-scale frequency cepstral coefficients feature of a target speech, and a mel-scale frequency cepstral coefficients feature of a source speech.
12. A training method for a voice conversion system, applicable to the voice conversion system according to claim 1 , wherein a training method for the neural network vocoder comprises:
inputting a mel-scale frequency cepstral coefficients feature of a target speech and the target speech to the neural network vocoder that is pre-trained; and
training the neural network vocoder by a method of Deep Transfer Learning.
13. The training method for the voice conversion system according to claim 12 , further comprising preprocessing the target speech, wherein the preprocessing comprises de-blanking and normalizing.
14. The training method for the voice conversion system according to claim 12 , further comprising extracting a parameter to obtain the mel-scale frequency cepstral coefficients feature of the multi-speaker speech recognition training corpus, a mel-scale frequency cepstral coefficients feature of a target speech, and a mel-scale frequency cepstral coefficients feature of a source speech.
15. A voice conversion method, characterized by comprising steps of:
by a speaker-independent automatic speech recognition model, converting a mel-scale frequency cepstral coefficients feature of an inputted source speech into a bottleneck feature of the source speech through the bottleneck layer, and outputting the bottleneck feature of the source speech to an Attention voice conversion network through the bottleneck layer;
where a training method for the speaker-independent automatic speech recognition model comprises:
inputting a number of a character encoding to which a word in a multi-speaker speech recognition training corpus is converted, together with a mel-scale frequency cepstral coefficients feature of the multi-speaker speech recognition training corpus, to the speaker-independent automatic speech recognition model; executing a backward propagation algorithm; and performing iterative optimization until the speaker-independent automatic speech recognition model is converged;
converting, by the Attention voice conversion network, the bottleneck feature of the source speech into a mel-scale frequency cepstral coefficients feature in conformity with a target speech; and
converting, by a neural network vocoder, the mel-scale frequency cepstral coefficients feature in conformity with the target speech into and output audio.
16. A terminal comprising the voice conversion system according to claim 1 .
17. A non-transitory computer readable storage medium, wherein a computer program is stored in the computer-readable storage medium, wherein the computer program is configured to, when executed, perform the method according to claim 2 .
18. A non-transitory computer-readable storage medium, wherein a computer program is stored in the non-transitory computer-readable storage medium, wherein the computer program is configured to, when executed, perform the method according to claim 6 .
19. A non-transitory computer-readable storage medium, wherein a computer program is stored in the non-transitory computer-readable storage medium, wherein the computer program is configured to, when executed, perform the method according to claim 12 .