Robust speech audio generation for video games
This specification describes a computer-implemented method of training a machine-learned speech audio generation system for use in video games. The training comprises: receiving one or more training examples. Each training example comprises: (i) ground-truth acoustic features for speech audio, (ii) speech content data representing speech content of the speech audio, and (iii) a ground-truth speaker identifier for a speaker of the speech audio. Parameters of the machine-learned speech audio generation system are updated to: (i) minimize a measure of difference between the predicted acoustic features of a training example and the corresponding ground-truth acoustic features of the training example, (ii) maximize a measure of difference between the first speaker classification for the training example and the corresponding ground-truth speaker identifier of the training example, and (iii) minimize a measure of difference between the second speaker classification for the training example and the corresponding ground-truth speaker identifier of the training example.
1. A computer-implemented method of training a machine-learned speech audio generation system for use in a video game, the training comprising:
receiving one or more training examples, each training example comprising: (i) ground-truth acoustic features for speech audio, (ii) speech content data representing speech content of the speech audio, and (iii) a ground-truth speaker identifier for a speaker of the speech audio;
for each of the one or more training examples:
generating, by a speaker encoder, a speaker embedding, comprising processing the ground-truth acoustic features;
generating, by an expression encoder, an expression embedding for the training example, comprising processing the ground-truth acoustic features;
classifying, by an expression-speaker classifier, the expression embedding to generate a first speaker classification;
generating, by a speech content encoder of a synthesizer, a speech content embedding, comprising processing the speech content data;
generating a combined embedding, comprising combining the speaker embedding, the expression embedding, and the speech content embedding;
classifying, by a combined-speaker classifier, the combined embedding to generate a second speaker classification;
decoding, by a decoder of the synthesizer, the combined embedding, to generate predicted acoustic features for the training example; and
updating parameters of the machine-learned speech audio generation system to: (i) minimize a measure of difference between the predicted acoustic features of a training example and the corresponding ground-truth acoustic features of the training example, (ii) maximize a measure of difference between the first speaker classification for the training example and the corresponding ground-truth speaker identifier of the training example, and (iii) minimize a measure of difference between the second speaker classification for the training example and the corresponding ground-truth speaker identifier of the training example.
2. The method of claim 1 , further comprising, for each of the one or more training examples:
classifying, by a content-speaker classifier, the speech content embedding to generate a third speaker classification, wherein the parameters of the machine-learned speech audio generation system are further updated to maximize a measure of difference between the third speaker classification for a training example and the corresponding ground-truth speaker identifier of the training example.
3. The method of claim 1 , wherein updating parameters of the machine-learned speech audio generation system comprises updating parameters of each of the expression encoder, the speaker encoder, and the synthesizer to: (i) minimize a measure of difference between the predicted acoustic features of a training example and the corresponding ground-truth acoustic features of the training example, and (ii) minimize a measure of difference between the second speaker classification for the training example and the corresponding ground-truth speaker identifier of the training example.
4. The method of claim 1 , wherein updating parameters of the machine-learned speech audio generation system comprises updating parameters of the expression encoder to maximize a measure of difference between the first speaker classification for the training example and the corresponding ground-truth speaker identifier of the training example.
5. The method of claim 2 , wherein updating parameters of the machine-learned speech audio generation system comprises updating parameters of the speech content encoder of the synthesizer to maximize a measure of difference between the third speaker classification for the training example and the corresponding ground-truth speaker identifier of the training example.
6. The method of claim 1 , wherein classifying, by the expression-speaker classifier, the expression embedding to generate a first speaker classification comprises processing the expression embedding using a gradient reversal layer that multiplies a gradient of an expression-speaker classification loss by a negative value when updating parameters of the machine-learned speech audio generation system, wherein the expression-speaker classification loss measures a difference between the first speaker classification for the training example and the corresponding ground-truth speaker identifier of the training example.
7. The method of claim 2 , wherein classifying, by the content-speaker classifier, the speech content embedding to generate the third speaker classification comprises processing the speech content embedding using a gradient reversal layer that multiplies a gradient of a content-speaker classification loss by a negative value when updating parameters of the machine-learned speech audio generation system, wherein the content-speaker classification loss measures a difference between the third speaker classification for a training example and the corresponding ground-truth speaker identifier of the training example.
8. The method of claim 1 , wherein the speaker encoder comprises a pre-trained portion and an adaptive portion which is being trained.
9. The method of claim 1 , wherein each training example further comprises ground-truth speech audio, the method comprising:
receiving the predicted acoustic features for a training example;
generating, by a vocoder, generated speech audio, comprising processing the predicted acoustic features; and
updating parameters of the vocoder to minimize a measure of difference between the generated speech audio and the corresponding ground-truth speech audio.
10. The method of claim 1 , wherein one or more of the speaker encoder, the expression encoder, and the synthesizer comprises neural network layers.
11. The method of claim 9 , wherein the vocoder comprises neural network layers.
12. The method of claim 1 , wherein the synthesizer comprises an attention mechanism.
13. The method of claim 1 , wherein the predicted acoustic features comprise amplitude, frequency and phase information for generated speech audio.
14. The method of claim 1 , wherein the predicted acoustic features comprise spectrogram parameters.
15. A computing system to train a machine-learned speech audio generation system to generate predicted acoustic features for generated speech audio for use in a video game, wherein the computing system is configured to:
receive one or more training examples, each training example comprising: (i) ground-truth acoustic features for speech audio, (ii) speech content data representing speech content of the speech audio, and (iii) a ground-truth speaker identifier for a speaker of the speech audio;
for each of the one or more training examples:
generate, by a speaker encoder, a speaker embedding, comprising processing the ground-truth acoustic features;
generate, by an expression encoder, an expression embedding for the training example, comprising processing the ground-truth acoustic features;
classify, by an expression-speaker classifier, the expression embedding to generate a first speaker classification;
generate, by a speech content encoder of a synthesizer, a speech content embedding, comprising processing the speech content data;
generate a combined embedding, comprising combining the speaker embedding, the expression embedding, and the speech content embedding;
classify, by a combined-speaker classifier, the combined embedding to generate a second speaker classification;
decode, by a decoder of the synthesizer, the combined embedding, to generate predicted acoustic features for the training example; and
update parameters of the machine-learned speech audio generation system to: (i) minimize a measure of difference between the predicted acoustic features of a training example and the corresponding ground-truth acoustic features of the training example, (ii) maximize a measure of difference between the first speaker classification for the training example and the corresponding ground-truth speaker identifier of the training example, and (iii) minimize a measure of difference between the second speaker classification for the training example and the corresponding ground-truth speaker identifier of the training example.
16. The computing system of claim 15 , wherein the system is further configured to, for each of the one or more training examples:
classify, by a content-speaker classifier, the speech content embedding to generate a third speaker classification, wherein the parameters of the machine-learned speech audio generation system are further updated to maximize a measure of difference between the third speaker classification for a training example and the corresponding ground-truth speaker identifier of the training example.
17. The computing system of claim 15 , wherein each training example further comprises ground-truth speech audio, and the system is further configured to:
receive the predicted acoustic features for a training example;
generate, by a vocoder, generated speech audio, comprising processing the predicted acoustic features; and
update parameters of the vocoder to minimize a measure of difference between the generated speech audio and the corresponding ground-truth speech audio.
18. A non-transitory computer-readable medium storing instructions which, when executed by a processor, cause the processor to:
receive one or more training examples, each training example comprising: (i) ground-truth acoustic features for speech audio, (ii) speech content data representing speech content of the speech audio, and (iii) a ground-truth speaker identifier for a speaker of the speech audio;
for each of the one or more training examples:
generate, by a speaker encoder, a speaker embedding, comprising processing the ground-truth acoustic features;
generate, by an expression encoder, an expression embedding for the training example, comprising processing the ground-truth acoustic features;
classify, by an expression-speaker classifier, the expression embedding to generate a first speaker classification;
generate, by a speech content encoder of a synthesizer, a speech content embedding, comprising processing the speech content data;
generate a combined embedding, comprising combining the speaker embedding, the expression embedding, and the speech content embedding;
classify, by a combined-speaker classifier, the combined embedding to generate a second speaker classification;
decode, by a decoder of the synthesizer, the combined embedding, to generate predicted acoustic features for the training example; and
update parameters of a machine-learned speech audio generation system to: (i) minimize a measure of difference between the predicted acoustic features of a training example and the corresponding ground-truth acoustic features of the training example, (ii) maximize a measure of difference between the first speaker classification for the training example and the corresponding ground-truth speaker identifier of the training example, and (iii) minimize a measure of difference between the second speaker classification for the training example and the corresponding ground-truth speaker identifier of the training example.
19. The non-transitory computer-readable medium of claim 18 storing further instructions, which when executed by the processor, cause the processor to:
classify, by a content-speaker classifier, the speech content embedding to generate a third speaker classification, wherein the parameters of the machine-learned speech audio generation system are further updated to maximize a measure of difference between the third speaker classification for a training example and the corresponding ground-truth speaker identifier of the training example.
20. The non-transitory computer-readable medium of claim 18 , wherein each training example further comprises ground-truth speech audio, the non-transitory computer-readable medium storing further instructions, which when executed by the processor, cause the processor to:
receive the predicted acoustic features for a training example;
generate, by a vocoder, generated speech audio, comprising processing the predicted acoustic features; and
update parameters of the vocoder to minimize a measure of difference between the generated speech audio and the corresponding ground-truth speech audio.