Expressive speech audio generation for video games
This specification describes a computer-implemented method of training a machine-learned speech audio generation system to generate predicted acoustic features for generated speech audio for use in a video game. The training comprises receiving one or more training examples. Each training example comprises: (i) ground-truth acoustic features for speech audio, (ii) speech content data representing speech content of the speech audio, and (iii) speech expression data representing speech expression of the speech audio. Parameters of the machine-learned speech audio generation system are updated by: (i) minimizing a measure of difference between the predicted acoustic features for a training example and the corresponding ground-truth acoustic features of the training example, and (ii) minimizing a measure of difference between the predicted prosodic features for the training example and the corresponding ground-truth prosodic features for the training example.
1. A computer-implemented method of training a machine-learned speech audio generation system to generate predicted acoustic features for generated speech audio for use in a video game, the training comprising:
receiving one or more training examples, each training example comprising: (i) ground-truth acoustic features for speech audio, (ii) speech content data representing speech content of the speech audio, and (iii) speech expression data representing speech expression of the speech audio;
for each of the one or more training examples:
generating an expression embedding, comprising processing the speech expression data;
generating a speech content embedding, comprising processing the speech content data;
generating predicted acoustic features for the training example using at least the expression embedding and the speech content embedding;
generating predicted prosodic features and ground-truth prosodic features, wherein generating the predicted prosodic features comprises processing the predicted acoustic features, and wherein generating the ground-truth prosodic features comprises processing the ground-truth acoustic features; and
updating parameters of the machine-learned speech audio generation system based on at least (i) a measure of difference between the predicted acoustic features for a training example and the corresponding ground-truth acoustic features of the training example, and (ii) a measure of difference between the predicted prosodic features for the training example and the corresponding ground-truth prosodic features for the training example.
2. The method of claim 1 , wherein the predicted prosodic features, and the ground-truth prosodic features, are generated by a machine-learning model that has been trained to extract prosodic features from acoustic features.
3. The method of claim 2 , wherein the machine-learning model is a neural network comprising an attention mechanism.
4. The method of claim 2 , wherein the machine-learning model has been trained to extract prosodic features relating to pitch and/or volume of speech audio represented by the acoustic features.
5. The method of claim 2 , wherein the expression embedding is generated by an expression encoder, and wherein updating parameters of the machine-learned speech audio generation system comprises updating parameters of at least the expression encoder while fixing parameters of the machine-learning model that has been trained to extract prosodic features from acoustic features.
6. The method of claim 1 , wherein an expression encoder comprises a portion of a pre-trained language model.
7. The method of claim 6 , wherein the portion of the pre-trained language model comprises one or more neural network layers of a pre-trained neural network.
8. The method of claim 7 , wherein the one or more neural network layers of the pre-trained neural network comprise a transformer block of a transformer-based neural network, wherein the transformer block is configured to perform self-attention on input tokens derived from the speech expression data.
9. The method of claim 6 , wherein the expression encoder further comprises a trainable adaptive portion.
10. The method of claim 9 , wherein the trainable adaptive portion of the expression encoder comprises one or more neural network layers.
11. A computing system to train a machine-learned speech audio generation system to generate predicted acoustic features for generated speech audio for use in a video game, wherein the system is configured to:
receive one or more training examples, each training example comprising: (i) ground-truth acoustic features for speech audio, (ii) speech content data representing speech content of the speech audio, and (iii) speech expression data representing speech expression of the speech audio;
for each of the one or more training examples:
generate an expression embedding, comprising processing the speech expression data;
generate a speech content embedding, comprising processing the speech content data;
generate predicted acoustic features for the training example using at least the expression embedding and the speech content embedding;
generate predicted prosodic features and ground-truth prosodic features, wherein generating the predicted prosodic features comprises processing the predicted acoustic features, and wherein generating the ground-truth prosodic features comprises processing the ground-truth acoustic features; and
update parameters of the machine-learned speech audio generation system based on at least: (i) a measure of difference between the predicted acoustic features for a training example and the corresponding ground-truth acoustic features of the training example, and (ii) a measure of difference between the predicted prosodic features for the training example and the corresponding ground-truth prosodic features for the training example.
12. The computing system of claim 11 , wherein the expression embedding is generated by an expression encoder, and wherein updating parameters of the machine-learned speech audio generation system comprises updating parameters of at least the expression encoder.
13. The computing system of claim 11 , wherein the predicted prosodic features, and the ground-truth prosodic features, are generated by a machine-learning model that has been trained to extract prosodic features from acoustic features.
14. The computing system of claim 11 , wherein the expression embedding is generated by an expression encoder, the expression encoder comprising a portion of a pre-trained language model.
15. The computing system of claim 14 , wherein the expression encoder further comprises a trainable adaptive portion.
16. The computing system of claim 11 , wherein the predicted acoustic features comprise a representation of frequency, magnitude, and phase for generated speech audio.
17. A non-transitory computer-readable medium storing instructions, which when executed by a processor, cause the processor to:
receive one or more training examples, each training example comprising: (i) ground-truth acoustic features representing speech audio, (ii) speech content data representing speech content of the speech audio, and (iii) speech expression data representing speech expression of the speech audio;
for each of the one or more training examples:
generate an expression embedding, comprising processing the speech expression data;
generate a speech content embedding, comprising processing the speech content data;
generating predicted acoustic features for the training example using at least the expression embedding and the speech content embedding;
generate predicted prosodic features and ground-truth prosodic features, wherein generating the predicted prosodic features comprises processing the predicted acoustic features, and wherein generating the ground-truth prosodic features comprises processing the ground-truth acoustic features; and
update parameters of a machine-learned speech audio generation system based on at least: (i) a measure of difference between the predicted acoustic features for a training example and the corresponding ground-truth acoustic features of the training example, and (ii) a measure of difference between the predicted prosodic features for the training example and the corresponding ground-truth prosodic features for the training example.
18. The non-transitory computer-readable medium of claim 17 , wherein the expression embedding is generated by an expression encoder, and wherein updating parameters of the machine-learned speech audio generation system comprises updating parameters of at least the expression encoder.
19. The non-transitory computer-readable medium of claim 17 , wherein the expression embedding is generated by an expression encoder, and wherein the expression encoder comprises a portion of a pre-trained language model.
20. The non-transitory computer-readable medium of claim 17 , wherein the prosodic features comprise features relating to pitch and/or volume of speech audio represented by the acoustic features.