Speech translation with performance characteristics

1 . A computer-implemented method comprising:

receiving first audio data representing first speech spoken by a first speaker in a source language, the first speech to be translated into synthesized speech in a target language different from the source language;

receiving first speaker embedding data representing first voice identity characteristics of a second speaker different from the first speaker;

processing the first audio data using a first encoder to generate first vocal performance embedding data representing first vocal performance characteristics of the first speech;

receiving first data representing a first transcript of the first speech in the source language;

performing machine translation of the first data to generate second data representing a second transcript in the target language, the second transcript corresponding to a translation of the first transcript;

receiving language embedding data representing language-dependent speech characteristics of the target language;

generating, using a second encoder, the first vocal performance embedding data, the language embedding data, and the second data, first transcript embedding data corresponding to the second transcript in the target language and the first vocal performance characteristics;

processing the first speaker embedding data and the first transcript embedding data using a first neural network to generate first acoustic embedding data corresponding to the synthesized speech, the first vocal performance characteristics, and the first voice identity characteristics, the first neural network implementing a first transformation representing an inverse of a second transformation implemented by a second neural network trained to suppress voice identity characteristics; and

processing the first acoustic embedding data using a decoder and the first speaker embedding data to generate second audio data representing the synthesized speech in the target language, the synthesized speech having the first voice identity characteristics and the first vocal performance characteristics.

2 . The computer-implemented method of claim 1 , further comprising:

receiving third audio data representing sample speech from a training dataset;

processing the third audio data using a third encoder to generate second speaker embedding data representing second voice identity characteristics of a third speaker of the sample speech;

processing the third audio data using a fourth encoder and the second speaker embedding data to generate second acoustic embedding data representing the sample speech with the second voice identity characteristics retained;

processing the second acoustic embedding data using a third neural network and the second speaker embedding data to generate third data representing the sample speech with the second voice identity characteristics suppressed;

determining fourth data representing a third transcript of the sample speech; and

training the third neural network using the third data and the fourth data to determine the second neural network, the second neural network representing an updated version of the third neural network.

3 . The computer-implemented method of claim 1 , further comprising:

processing the first audio data using a first component to generate third audio data representing the first audio data with at least a portion of noise content removed;

processing the third audio data using the first encoder to generate second vocal performance embedding data;

processing fourth audio data using the first component to generate fifth audio data representing a noise content of the fourth audio data, the fourth audio data representing speech recorded in a low-noise environment;

processing the fifth audio data using a third encoder to generate noise embedding data; and

determining the first vocal performance embedding data using the second vocal performance embedding data and the noise embedding data.

4 . The computer-implemented method of claim 1 , further comprising:

determining that the second data corresponds to a first duration;

receiving duration data indicating that the first speech corresponds to a second duration different from the first duration; and

generating, using the first transcript embedding data and the duration data, second transcript embedding data corresponding to the second duration, wherein generating the first acoustic embedding data includes processing the second transcript embedding data using the first neural network.

5 . A computer-implemented method comprising:

receiving first audio data representing first speech in a first language;

receiving first data representing first voice identity characteristics for synthesizing second speech;

determining, using the first audio data, second data representing vocal performance characteristics of the first speech;

receiving third data representing a transcript of the second speech in a target language, the second speech representing a translation of the first speech;

determining, using the third data and the second data, fourth data representing transcript embeddings corresponding to the transcript of the second speech and the vocal performance characteristics of the first speech;

processing the fourth data and the first data using a first neural network to generate fifth data representing acoustic embeddings corresponding to the second speech, the vocal performance characteristics, and the first voice identity characteristics, the first neural network implementing a first transformation representing an inverse of a second transformation trained to suppress voice identity characteristics; and

processing the fifth data using a decoder to determine second audio data representing the second speech, the second speech having the first voice identity characteristics and the vocal performance characteristics.

6 . The computer-implemented method of claim 5 , further comprising:

receiving third audio data representing third speech;

determining first embedding data representing second voice identity characteristics of the third speech;

determining, using the third audio data and the first embedding data, second embedding data representing the third speech and the second voice identity characteristics;

processing the second embedding data using a third transformation and the first embedding data to generate third embedding data representing the third speech with the second voice identity characteristics suppressed;

determining fifth data representing a transcript of the third speech;

training the third transformation using the third embedding data and the fifth data to determine the second transformation, the second transformation representing an updated version of the third transformation; and

determining the first transformation using the second transformation.

7 . The computer-implemented method of claim 5 , further comprising:

processing the first audio data using a first component to generate third audio data representing the first audio data with at least a portion of noise content removed;

determining, using the third audio data, fifth data representing second vocal performance characteristics of the first speech;

receiving fourth audio data representing a speech recorded in a low-noise environment;

processing the fourth audio data using the first component to generate fifth audio data representing a noise content of the fourth audio data;

determining, using the fifth audio data, sixth data representing a noise embedding; and

determining the second data using the fifth data and the sixth data.

8 . The computer-implemented method of claim 5 , further comprising:

determining that the third data corresponds to a first duration; and

receiving duration data indicating that the first speech corresponds to a second duration different from the first duration, wherein generating the fourth data includes generating the fourth data to correspond to the second duration.

9 . The computer-implemented method of claim 5 , further comprising:

processing the first audio data to determine fifth data representing a transcript of the first speech;

performing machine translation of the fifth data to generate sixth data representing a translation of the first speech into the target language; and

performing grapheme-to-phoneme processing of the sixth data to determine the third data.

10 . The computer-implemented method of claim 5 , further comprising:

receiving embedding data representing speech characteristics corresponding to the target language, wherein determining the fourth data includes using the embedding data and the fourth data additionally corresponds to the speech characteristics of the target language.

11 . The computer-implemented method of claim 5 , further comprising:

receiving first multimedia content including video data and the first audio data;

determining first timing data representing a first temporal alignment between the first audio data and the video data;

determining, using the first timing data, second timing data representing a second temporal alignment between the second audio data and the video data; and

generating, using the second timing data, the video data, and the second audio data, second media content representing the video data dubbed with the second audio data.

12 . The computer-implemented method of claim 5 , further comprising:

receiving third audio data representing a plurality of speech samples;

processing the third audio data using an encoder to determine sixth embedding data representing a distribution of representations the plurality of speech samples in a latent space;

processing the sixth embedding data using the decoder to generate fourth audio data representing a reconstruction of the plurality of speech samples;

determining, using the sixth embedding data, a regularization loss term;

determining, using the fourth audio data, a reconstruction loss term; and

training the encoder and the decoder using the regularization loss term and the reconstruction loss term.

13 . A system, comprising:

at least one processor; and

at least one memory comprising instructions that, when executed by the at least one processor, cause the system to:

receive first audio data representing first speech in a first language;

receive first data representing first voice identity characteristics for synthesizing second speech;

determine, using the first audio data, second data representing vocal performance characteristics of the first speech;

receive third data representing a transcript of the second speech in a target language, the second speech representing a translation of the first speech;

determine, using the third data and the second data, fourth data representing transcript embeddings corresponding to the transcript of the second speech and the vocal performance characteristics of the first speech;

process the fourth data and the first data using a first neural network to generate fifth data representing acoustic embeddings corresponding to the second speech, the vocal performance characteristics, and the first voice identity characteristics, the first neural network implementing a first transformation representing an inverse of a second transformation trained to suppress voice identity characteristics; and

process the fifth data using a decoder to determine second audio data representing the second speech, the second speech having the first voice identity characteristics and the vocal performance characteristics.

14 . The system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

receive third audio data representing third speech;

determine first embedding data representing second voice identity characteristics of the third speech;

determine, using the third audio data and the first embedding data, second embedding data representing the third speech and the second voice identity characteristics;

process the second embedding data using a third transformation and the first embedding data to generate third embedding data representing the third speech with the second voice identity characteristics suppressed;

determine fifth data representing a transcript of the third speech;

train the third transformation using the third embedding data and the fifth data to determine the second transformation, the second transformation representing an updated version of the third transformation; and

determine the first transformation using the second transformation.

15 . The system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

process the first audio data using a first component to generate third audio data representing the first audio data with at least a portion of noise content removed;

determine, using the third audio data, fifth data representing second vocal performance characteristics of the first speech;

receive fourth audio data representing a speech recorded in a low-noise environment;

process the fourth audio data using the first component to generate fifth audio data representing a noise content of the fourth audio data;

determine, using the fifth audio data, sixth data representing a noise embedding; and

determine the second data using the fifth data and the sixth data.

16 . The system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

determine that the third data corresponds to a first duration; and

receive duration data indicating that the first speech corresponds to a second duration different from the first duration, wherein generating the fourth data includes generating the fourth data to correspond to the second duration.

17 . The system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

process the first audio data to determine fifth data representing a transcript of the first speech;

perform machine translation of the fifth data to generate sixth data representing a translation of the first speech into the target language; and

perform grapheme-to-phoneme processing of the sixth data to determine third first data.

18 . The system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

receive embedding data representing speech characteristics corresponding to the target language, wherein determining the fourth data includes using the embedding data and the fourth data additionally corresponds to the speech characteristics of the target language.

19 . The system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

receive first multimedia content including video data and the first audio data;

determine first timing data representing a first temporal alignment between the first audio data and the video data;

determine, using the first timing data, second timing data representing a second temporal alignment between the second audio data and the video data; and

generate, using the second timing data, the video data, and the second audio data, second media content representing the video data dubbed with the second audio data.

20 . The system of claim 13 , wherein the at least one memory further comprises instructions that, when executed by the at least one processor, further cause the system to:

receive third audio data representing a plurality of speech samples;

process the third audio data using an encoder to determine sixth embedding data representing a distribution of representations the plurality of speech samples in a latent space;

process the sixth embedding data using the decoder to generate fourth audio data representing a reconstruction of the plurality of speech samples;

determine, using the sixth embedding data, a regularization loss term;

determine, using the fourth audio data, a reconstruction loss term; and

train the encoder and the decoder using the regularization loss term and the reconstruction loss term.