Generating audio using neural networks

1. A neural network system implemented by one or more computers, wherein the neural network system is configured to autoregressively generate an output sequence of audio data that comprises a respective audio sample at each of a plurality of time steps, wherein the output sequence of audio data is a verbalization of a text segment, and

wherein the neural network system comprises:

a convolutional subnetwork comprising one or more audio-processing convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of time steps:

receive: (i) a current sequence of audio data that comprises the respective audio sample at each time step that precedes the time step in the output sequence, and (ii) features of the text segment, and

process the current sequence of audio data and the features of the text segment to generate an alternative representation for the time step; and

an output layer, wherein the output layer is configured to, for each of the plurality of time steps:

receive the alternative representation for the time step, and

process the alternative representation for the time step to generate an output that defines a score distribution over a plurality of possible audio samples for the time step.

2. The neural network system of claim 1 , wherein the one or more computers are included in a mobile device.

3. The neural network system of claim 1 , wherein the one or more computers are included in a personal digital assistant device.

4. The neural network system of claim 1 , wherein the neural network system further comprises:

a subsystem configured to, for each of the plurality of time steps:

select an audio sample at the time step in the output sequence in accordance with the score distribution for the time step.

5. The neural network system of claim 4 , wherein selecting the audio value comprises:

sampling from the score distribution.

6. The neural network system of claim 4 , wherein selecting the audio value comprises:

selecting an audio sample having a highest score according to the score distribution.

7. The neural network system of claim 1 , wherein each of the plurality of time steps corresponds to a respective time in an audio waveform, and wherein the respective audio sample at each of the plurality of time steps is an amplitude value of the audio waveform at the corresponding time.

8. The neural network system of claim 1 , wherein each of the plurality of time steps corresponds to a respective time in an audio waveform, and wherein the respective audio sample at each of the plurality of time steps is a compressed or a companded representation of the audio waveform at the corresponding time.

9. The neural network system of claim 1 , wherein the audio-processing convolutional neural network layers are causal convolutional neural network layers.

10. The neural network system of claim 1 , wherein the audio-processing convolutional neural network layers include one or more dilated convolutional neural network layers.

11. The neural network system of claim 10 , wherein the audio-processing convolutional neural network layers include multiple blocks of dilated convolutional neural network layers, wherein each block comprises multiple dilated convolutional neural network layers with increasing dilation.

12. The neural network system of claim 1 , wherein one or more of the audio-processing convolutional neural network layers have gated activation units.

13. The neural network system of claim 1 , wherein, at each of the plurality of time steps, the alternative representation is conditioned on a neural network input.

14. The neural network system of claim 13 , wherein the neural network input comprises one or more of: intonation pattern values, speaker identity information, language identity information, and speaking style information.

15. The neural network system of claim 1 , wherein the convolutional subnetwork comprises residual connections, skip connections, or both.

16. The neural network system of claim 1 , wherein processing the current sequence of audio data and the features of the text segment to generate an alternative representation for the time step comprises reusing values computed for previous time steps.

17. One or more non-transitory computer-readable storage media encoded with instructions that when executed by one or more computers cause the one or more computers to implement a neural network system,

wherein the neural network system is configured to autoregressively generate an output sequence of audio data that comprises a respective audio sample at each of a plurality of time steps, wherein the output sequence of audio data is a verbalization of a text segment, and

wherein the neural network system comprises:

a convolutional subnetwork comprising one or more audio-processing convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of time steps:

receive: (i) a current sequence of audio data that comprises the respective audio sample at each time step that precedes the time step in the output sequence, and (ii) features of the text segment, and

process the current sequence of audio data and the features of the text segment to generate an alternative representation for the time step; and

an output layer, wherein the output layer is configured to, for each of the plurality of time steps:

receive the alternative representation for the time step, and

process the alternative representation for the time step to generate an output that defines a score distribution over a plurality of possible audio samples for the time step.

18. The non-transitory computer-readable storage media of claim 17 , wherein the one or more computers are included in a mobile device.

19. The non-transitory computer-readable storage media of claim 17 , wherein the one or more computers are included in a personal digital assistant device.

20. The non-transitory computer-readable storage media of claim 17 , wherein the neural network system further comprises:

a subsystem configured to, for each of the plurality of time steps:

select an audio sample at the time step in the output sequence in accordance with the score distribution for the time step.

21. The non-transitory computer-readable storage media of claim 17 , wherein each of the plurality of time steps corresponds to a respective time in an audio waveform, and wherein the respective audio sample at each of the plurality of time steps is an amplitude value of the audio waveform at the corresponding time.

22. The non-transitory computer-readable storage media of claim 17 , wherein the audio-processing convolutional neural network layers are causal convolutional neural network layers.

23. The non-transitory computer-readable storage media of claim 17 , wherein the audio-processing convolutional neural network layers include one or more dilated convolutional neural network layers.

24. A method, performed by one or more computers, of autoregressively generating an output sequence of audio data that comprises a respective audio sample at each of a plurality of time steps,

wherein the output sequence of audio data is a verbalization of a text segment,

wherein the method comprises, for each of the plurality of time steps:

providing a current sequence of audio data and features of the text segment as input to a convolutional subnetwork comprising one or more audio-processing convolutional neural network layers,

wherein the current sequence comprises the respective audio sample at each time step that precedes the time step in the output sequence, and

wherein the convolutional subnetwork is configured to, for each of the plurality of time steps:

receive the current sequence of audio data and the features of the text segment, and

process the current sequence of audio data and the features of the text segment to generate an alternative representation for the time step; and

providing the alternative representation for the time step as input to an output layer, wherein the output layer is configured to, for each of the plurality of time steps:

receive the alternative representation for the time step, and

process the alternative representation for the time step to generate an output that defines a score distribution over a plurality of possible audio samples for the time step.

25. The method of claim 24 , wherein the one or more computers are included in a mobile device.

26. The method of claim 24 , wherein the one or more computers are included in a personal digital assistant device.

27. The method of claim 24 , further comprising:

providing the score distribution for the time step as input to a subsystem, wherein the subsystem is configured to, for each of the plurality of time steps:

select an audio sample at the time step in the output sequence in accordance with the score distribution for the time step.

28. The method of claim 24 , wherein each of the plurality of time steps corresponds to a respective time in an audio waveform, and wherein the respective audio sample at each of the plurality of time steps is an amplitude value of the audio waveform at the corresponding time.

29. The method of claim 24 , wherein the audio-processing convolutional neural network layers are causal convolutional neural network layers.

30. The method of claim 24 , wherein the audio-processing convolutional neural network layers include one or more dilated convolutional neural network layers.