One stage text-to-speech system with prosody control
This specification describes a computer-implemented method of generating expressive speech audio for use in a video game. The method comprises receiving user-input text data and receiving prosody data obtained from user-input speech style data. The user-input text data is processed by a text encoder to generate a text feature embedding. A speech feature embedding is generated by an embedding converter based upon the text feature embedding. The prosody data is processing by a prosody encoder to generate a prosody feature embedding. A speech signal is generated by a speech decoder based upon the speech feature embedding and the prosody feature embedding.
1 . A computer-implemented method of training a machine-learned expressive speech audio generation system for use in a video game, the method comprising:
receiving one or more training examples, each training example comprising: (i) a reference speech signal and (ii) reference text data corresponding to the reference speech signal;
for each of the one or more training examples:
receiving prosody data corresponding to the reference speech signal;
processing, by a speech encoder, the reference speech signal to generate a speech feature embedding;
processing, by a text encoder, the reference text data to generate a text feature embedding;
generating, by an embedding converter, one or more embedding conversions, wherein generating comprises:
processing, by the embedding converter, the speech feature embedding to generate a speech converted text feature embedding; and/or
processing, by the embedding converter, the text feature embedding to generate a text converted speech feature embedding;
the one or more embedding conversions comprising the speech converted text feature embedding and/or the text converted speech feature embedding;
processing, by a prosody encoder, the prosody data to generate a prosody feature embedding;
generating, by a speech decoder, a reconstructed speech signal based upon the speech feature embedding and the prosody feature embedding;
determining a loss value using a loss function based upon the reconstructed speech signal, the reference speech signal and the one or more embedding conversions, wherein the loss function comprises a conversion loss term based upon an error measure of the generated one or more embedding conversions; and
updating one or more of the following: the speech decoder, the prosody encoder, the speech encoder, the embedding converter, the speech encoder, or the text encoder based upon the determined loss value.
2 . The method of claim 1 , wherein the loss function comprises a reconstruction loss term based upon a measure of difference between the reconstructed speech signal and the reference speech signal.
3 . The method of claim 1 , wherein the error measure is based upon a KL-divergence.
4 . The method of claim 1 , wherein the loss function comprises an adversarial loss term based upon an ability of a discriminator machine-learning model to differentiate between the reconstructed speech signal and the reference speech signal.
5 . The method of claim 1 , further comprising:
generating, by a prosody analyzer, the prosody data from the reference speech signal.
6 . The method of claim 1 , wherein the prosody data comprises prosodic statistical features.
7 . The method of claim 6 , wherein the prosody data comprises one or more statistical features of a pitch contour and/or a volume contour for the reference speech signal.
8 . The method of claim 1 , wherein the updating based upon the determined loss value comprises using a gradient-based optimization technique.
9 . The method of claim 1 , wherein the loss function is based upon an evidence-based lower bound (ELBO).
10 . The method of claim 1 , further comprising:
processing, by a duration predictor, the text feature embedding to generate a predicted spoken duration for the reference text data; and
modifying the text feature embedding based upon the predicted spoken duration.
11 . A computing system for training a machine-learned expressive speech audio generation system for use in a video game, wherein the system is configured to:
receive one or more training examples, each training example comprising: (i) a reference speech signal and (ii) reference text data corresponding to the reference speech signal;
for each of the one or more training examples:
receive prosody data corresponding to the reference speech signal;
process, by a speech encoder, the reference speech signal to generate a speech feature embedding;
process, by a text encoder, the reference text data to generate a text feature embedding;
generate, by an embedding converter, one or more embedding conversions, wherein, in generating, the system is configured to:
process, by the embedding converter, the speech feature embedding to generate a speech converted text feature embedding; and/or
process, by the embedding converter, the text feature embedding to generate a text converted speech feature embedding;
the one or more embedding conversions comprising the speech converted text feature embedding and/or the text converted speech feature embedding;
process, by a prosody encoder, the prosody data to generate a prosody feature embedding;
generate, by a speech decoder, a reconstructed speech signal based upon the speech feature embedding and the prosody feature embedding;
determine a loss value using a loss function based upon the reconstructed speech signal, the reference speech signal and the one or more embedding conversions, wherein the loss function comprises a conversion loss term based upon an error measure of the generated one or more embedding conversions; and
update one or more of the following: the speech decoder, the prosody encoder, the speech encoder, the embedding converter, the speech encoder, or the text encoder based upon the determined loss value.
12 . The system of claim 11 , wherein the loss function comprises a reconstruction loss term based upon a measure of difference between the reconstructed speech signal and the reference speech signal.
13 . The system of claim 11 , wherein the error measure is based upon a KL-divergence.
14 . The system of claim 11 , wherein the loss function comprises an adversarial loss term based upon an ability of a discriminator machine-learning model to differentiate between the reconstructed speech signal and the reference speech signal.
15 . The system of claim 11 , wherein the system is further configured to:
generate, by a prosody analyzer, the prosody data from the reference speech signal.
16 . The system of claim 11 , wherein the prosody data comprises prosodic statistical features.
17 . The system of claim 11 , wherein the prosody data comprises one or more statistical features of a pitch contour and/or a volume contour for the reference speech signal.
18 . The system of claim 11 , wherein the updating based upon the determined loss value comprises using a gradient-based optimization technique.
19 . The system of claim 11 , wherein the loss function is based upon an evidence-based lower bound (ELBO).
20 . One or more non-transitory computer-readable storage media comprising instructions which, when executed by one or more processors, cause the one or more processors to carry out a method comprising:
receiving one or more training examples, each training example comprising: (i) a reference speech signal and (ii) reference text data corresponding to the reference speech signal;
for each of the one or more training examples:
receiving prosody data corresponding to the reference speech signal;
processing, by a speech encoder, the reference speech signal to generate a speech feature embedding;
processing, by a text encoder, the reference text data to generate a text feature embedding;
generating, by an embedding converter, one or more embedding conversions, wherein generating comprises:
processing, by the embedding converter, the speech feature embedding to generate a speech converted text feature embedding; and/or
processing, by the embedding converter, the text feature embedding to generate a text converted speech feature embedding;
the one or more embedding conversions comprising the speech converted text feature embedding and/or the text converted speech feature embedding;
processing, by a prosody encoder, the prosody data to generate a prosody feature embedding;
generating, by a speech decoder, a reconstructed speech signal based upon the speech feature embedding and the prosody feature embedding;
determining a loss value using a loss function based upon the reconstructed speech signal, the reference speech signal and the one or more embedding conversions, wherein the loss function comprises a conversion loss term based upon an error measure of the generated one or more embedding conversions; and
updating one or more of the following: the speech decoder, the prosody encoder, the speech encoder, the embedding converter, the speech encoder, or the text encoder based upon the determined loss value.