One stage text-to-speech system with prosody control

1 . A computer-implemented method of training a machine-learned expressive speech audio generation system for use in a video game, the method comprising:

receiving one or more training examples, each training example comprising: (i) a reference speech signal and (ii) reference text data corresponding to the reference speech signal;

for each of the one or more training examples:

receiving prosody data corresponding to the reference speech signal;

processing, by a speech encoder, the reference speech signal to generate a speech feature embedding;

processing, by a text encoder, the reference text data to generate a text feature embedding;

generating, by an embedding converter, one or more embedding conversions, wherein generating comprises:

processing, by the embedding converter, the speech feature embedding to generate a speech converted text feature embedding; and/or

processing, by the embedding converter, the text feature embedding to generate a text converted speech feature embedding;

the one or more embedding conversions comprising the speech converted text feature embedding and/or the text converted speech feature embedding;

processing, by a prosody encoder, the prosody data to generate a prosody feature embedding;

generating, by a speech decoder, a reconstructed speech signal based upon the speech feature embedding and the prosody feature embedding;

determining a loss value using a loss function based upon the reconstructed speech signal, the reference speech signal and the one or more embedding conversions, wherein the loss function comprises a conversion loss term based upon an error measure of the generated one or more embedding conversions; and

updating one or more of the following: the speech decoder, the prosody encoder, the speech encoder, the embedding converter, the speech encoder, or the text encoder based upon the determined loss value.

2 . The method of claim 1 , wherein the loss function comprises a reconstruction loss term based upon a measure of difference between the reconstructed speech signal and the reference speech signal.

3 . The method of claim 1 , wherein the error measure is based upon a KL-divergence.

4 . The method of claim 1 , wherein the loss function comprises an adversarial loss term based upon an ability of a discriminator machine-learning model to differentiate between the reconstructed speech signal and the reference speech signal.

5 . The method of claim 1 , further comprising:

generating, by a prosody analyzer, the prosody data from the reference speech signal.

6 . The method of claim 1 , wherein the prosody data comprises prosodic statistical features.

7 . The method of claim 6 , wherein the prosody data comprises one or more statistical features of a pitch contour and/or a volume contour for the reference speech signal.

8 . The method of claim 1 , wherein the updating based upon the determined loss value comprises using a gradient-based optimization technique.

9 . The method of claim 1 , wherein the loss function is based upon an evidence-based lower bound (ELBO).

10 . The method of claim 1 , further comprising:

processing, by a duration predictor, the text feature embedding to generate a predicted spoken duration for the reference text data; and

modifying the text feature embedding based upon the predicted spoken duration.

11 . A computing system for training a machine-learned expressive speech audio generation system for use in a video game, wherein the system is configured to:

receive one or more training examples, each training example comprising: (i) a reference speech signal and (ii) reference text data corresponding to the reference speech signal;

for each of the one or more training examples:

receive prosody data corresponding to the reference speech signal;

process, by a speech encoder, the reference speech signal to generate a speech feature embedding;

process, by a text encoder, the reference text data to generate a text feature embedding;

generate, by an embedding converter, one or more embedding conversions, wherein, in generating, the system is configured to:

process, by the embedding converter, the speech feature embedding to generate a speech converted text feature embedding; and/or

process, by the embedding converter, the text feature embedding to generate a text converted speech feature embedding;

the one or more embedding conversions comprising the speech converted text feature embedding and/or the text converted speech feature embedding;

process, by a prosody encoder, the prosody data to generate a prosody feature embedding;

generate, by a speech decoder, a reconstructed speech signal based upon the speech feature embedding and the prosody feature embedding;

determine a loss value using a loss function based upon the reconstructed speech signal, the reference speech signal and the one or more embedding conversions, wherein the loss function comprises a conversion loss term based upon an error measure of the generated one or more embedding conversions; and

update one or more of the following: the speech decoder, the prosody encoder, the speech encoder, the embedding converter, the speech encoder, or the text encoder based upon the determined loss value.

12 . The system of claim 11 , wherein the loss function comprises a reconstruction loss term based upon a measure of difference between the reconstructed speech signal and the reference speech signal.

13 . The system of claim 11 , wherein the error measure is based upon a KL-divergence.

14 . The system of claim 11 , wherein the loss function comprises an adversarial loss term based upon an ability of a discriminator machine-learning model to differentiate between the reconstructed speech signal and the reference speech signal.

15 . The system of claim 11 , wherein the system is further configured to:

generate, by a prosody analyzer, the prosody data from the reference speech signal.

16 . The system of claim 11 , wherein the prosody data comprises prosodic statistical features.

17 . The system of claim 11 , wherein the prosody data comprises one or more statistical features of a pitch contour and/or a volume contour for the reference speech signal.

18 . The system of claim 11 , wherein the updating based upon the determined loss value comprises using a gradient-based optimization technique.

19 . The system of claim 11 , wherein the loss function is based upon an evidence-based lower bound (ELBO).

20 . One or more non-transitory computer-readable storage media comprising instructions which, when executed by one or more processors, cause the one or more processors to carry out a method comprising:

receiving one or more training examples, each training example comprising: (i) a reference speech signal and (ii) reference text data corresponding to the reference speech signal;

for each of the one or more training examples:

receiving prosody data corresponding to the reference speech signal;

processing, by a speech encoder, the reference speech signal to generate a speech feature embedding;

processing, by a text encoder, the reference text data to generate a text feature embedding;

generating, by an embedding converter, one or more embedding conversions, wherein generating comprises:

processing, by the embedding converter, the speech feature embedding to generate a speech converted text feature embedding; and/or

processing, by the embedding converter, the text feature embedding to generate a text converted speech feature embedding;

the one or more embedding conversions comprising the speech converted text feature embedding and/or the text converted speech feature embedding;

processing, by a prosody encoder, the prosody data to generate a prosody feature embedding;

generating, by a speech decoder, a reconstructed speech signal based upon the speech feature embedding and the prosody feature embedding;

determining a loss value using a loss function based upon the reconstructed speech signal, the reference speech signal and the one or more embedding conversions, wherein the loss function comprises a conversion loss term based upon an error measure of the generated one or more embedding conversions; and

updating one or more of the following: the speech decoder, the prosody encoder, the speech encoder, the embedding converter, the speech encoder, or the text encoder based upon the determined loss value.