Synthetic speech processing

1 . A computer-implemented method for generating speech, the method comprising:

receiving, from at least one microphone of a user device, first audio data representing first speech corresponding to a first voice;

processing the first audio data to determine first Mel-spectrogram data representing the first speech;

receiving second audio data corresponding to a second voice;

processing the second audio data to determine second Mel-spectrogram data representing the second voice;

processing, using a first encoder, the first Mel-spectrogram data to determine first encoded data corresponding to phoneme characteristics of the first speech;

processing, using a second encoder, the first Mel-spectrogram data to determine second encoded data corresponding to a phrase of the first speech;

processing, using a third encoder, the second Mel-spectrogram data to determine third encoded data corresponding to vocal characteristics of the second voice;

processing, using a decoder, the first encoded data, the second encoded data, and the third encoded data to determine third Mel-spectrogram data corresponding to the phoneme characteristics, the vocal characteristics, and the phrase; and

processing, using a vocoder, the third Mel-spectrogram data to determine output audio data corresponding to the vocal characteristics of the second voice.

2 . The computer-implemented method of claim 1 , wherein the decoder is a trained neural network, further comprising:

receiving text data;

processing, using a fourth encoder, the text data to determine fourth encoded data corresponding to a second phrase;

processing, using the decoder, the first encoded data, the third encoded data, and the fourth encoded data to determine fourth Mel-spectrogram data corresponding to the phoneme characteristics, the vocal characteristics, and the second phrase; and

processing, using the vocoder, the fourth Mel-spectrogram data to determine second output audio data corresponding to the vocal characteristics of the second voice.

3 . A computer-implemented method comprising:

receiving, from at least one microphone of a user device, first audio data representing first noise and first speech corresponding to a first voice, the first speech including a first plurality of words;

receiving second audio data corresponding to a second voice different from the first voice;

processing, using a first encoder, the first audio data to determine first encoded data corresponding to phoneme characteristics of the first speech;

processing, using a second encoder, the first audio data to determine second encoded data corresponding to the first noise and to a phrase of the first speech;

processing, using a third encoder, the second audio data to determine third encoded data corresponding to vocal characteristics of the second voice; and

processing, using a decoder, the first encoded data, the second encoded data, and the third encoded data to determine third audio data representing second speech corresponding to the second voice, the second speech including the first plurality of words.

4 . The computer-implemented method of claim 3 , further comprising:

receiving fourth audio data representing third speech corresponding to the first voice;

processing, using the first encoder, the fourth audio data to determine fourth encoded data corresponding to second phoneme characteristics of the third speech;

processing, using the second encoder, the fourth audio data to determine fifth encoded data corresponding to a second phrase of the third speech; and

processing, using the decoder, the fourth encoded data and the fifth encoded data to determine fifth audio data corresponding to the second phoneme characteristics, the vocal characteristics, and the second phrase.

5 . The computer-implemented method of claim 3 , further comprising:

processing the third audio data using a vocoder to determine output audio data, wherein the third audio data corresponds to the phoneme characteristics, the vocal characteristics, and the phrase; and

causing output of the output audio data.

6 . The computer-implemented method of claim 3 , further comprising:

processing, using an automatic speech recognition component, the first audio data to determine text data representing the first speech;

sending, to a remote system, the text data and output audio data corresponding to the third audio data;

causing the remote system to output the text data and the output audio data; and

receiving, from the remote system, an indication that the text data corresponds to the first speech.

7 . The computer-implemented method of claim 3 , further comprising:

determining that a user profile includes an indication of sending the third audio data;

sending, from the user device to a remote system, output data corresponding to the third audio data; and

deleting, by the user device the first audio data.

8 . The computer-implemented method of claim 3 , further comprising:

determining fourth audio data representing an utterance;

determining fifth audio data representing second noise;

determining, using the fourth audio data and the fifth audio data, sixth audio data representing the utterance and the second noise;

processing, using the first encoder, the second encoder, and the decoder, the sixth audio data to determine seventh audio data; and

determining a difference between the seventh audio data and the fourth audio data.

9 . The computer-implemented method of claim 3 , further comprising:

determining text data corresponding to the first speech;

processing, using a fourth encoder, the text data to determine fourth encoded data corresponding to the phrase; and

determining a difference between the second encoded data and the fourth encoded data.

10 . The computer-implemented method of claim 3 , further comprising:

determining that the first audio data includes a representation of a wakeword;

determining text data corresponding to the first speech;

sending, to a remote system, the third audio data and the text data; and

receiving, from the remote system, an indication that the third audio data corresponds to the text data.

11 . The computer-implemented method of claim 3 , further comprising:

receiving, from the at least one microphone of the user device, fourth audio data; and

processing the fourth audio data to determine the first audio data, wherein the first audio data comprises a representation of a Mel-spectrogram.

12 . A system comprising:

at least one processor; and

at least one memory including instructions that, when executed by the at least one processor, cause the system to:

receive, from at least one microphone of a user device, first audio data representing first noise and first speech corresponding to a first voice, the first speech including a first plurality of words;

receive second audio data corresponding to a second voice different from the first voice;

process, using a first encoder, the first audio data to determine first encoded data corresponding to phoneme characteristics of the first speech;

process, using a second encoder, the first audio data to determine second encoded data corresponding to the first noise and to a phrase of the first speech;

process, using a third encoder, the second audio data to determine third encoded data corresponding to vocal characteristics of the second voice; and

process, using a decoder, the first encoded data, the second encoded data, and the third encoded data to determine third audio data representing second speech corresponding to the second voice, the second speech including the first plurality of words.

13 . The system of claim 12 , wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to:

receive fourth audio data representing third speech corresponding to the first voice;

process, using the first encoder, the fourth audio data to determine fourth encoded data corresponding to second phoneme characteristics of the third speech;

process, using the second encoder, the fourth audio data to determine fifth encoded data corresponding to a second phrase of the third speech; and

process, using the decoder, the fourth encoded data and the fifth encoded data to determine fifth audio data corresponding to the second phoneme characteristics, the vocal characteristics, and the second phrase.

14 . The system of claim 12 , wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to:

process the third audio data using a vocoder to determine output audio data, the third audio data corresponding to the phoneme characteristics, the vocal characteristics, and the phrase; and

cause output of the output audio data.

15 . The system of claim 12 , wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to:

process, using an automatic speech recognition component, the first audio data to determine text data representing the first speech;

send, to a remote system, the text data and output audio data corresponding to the third audio data;

cause the remote system to output the text data and the output audio data; and

receive, from the remote system, an indication that the text data corresponds to the first speech.

16 . The system of claim 12 , wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to:

determine that a user profile includes an indication of sending the third audio data;

send, from the user device to a remote system, output data corresponding to the third audio data; and

delete, by the user device the first audio data.

17 . The system of claim 12 , wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to:

determine fourth audio data representing an utterance;

determine fifth audio data representing second noise;

determine, using the fourth audio data and the fifth audio data, sixth audio data representing the utterance and the second noise;

process, using the first encoder, the second encoder, and the decoder, the sixth audio data to determine seventh audio data; and

determine a difference between the seventh audio data and the fourth audio data.

18 . The system of claim 12 , wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to:

determine text data corresponding to the first speech;

process, using a fourth encoder, the text data to determine fourth encoded data corresponding to the phrase; and

determine a difference between the second encoded data and the fourth encoded data.

19 . The system of claim 12 , wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to:

determine that the first audio data includes a representation of a wakeword;

determine text data corresponding to the first speech;

send, to a remote system, the third audio data and the text data; and

receive, from the remote system, an indication that the third audio data corresponds to the text data.

20 . The system of claim 12 , wherein the at least one memory further includes instructions that, when executed by the at least one processor, further cause the system to:

receive, from the at least one microphone of the user device, fourth audio data; and

process the fourth audio data to determine the first audio data, wherein the first audio data comprises a representation of a Mel-spectrogram.