Generating audio using neural networks
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating an output sequence of audio data that comprises a respective audio sample at each of a plurality of time steps. One of the methods includes, for each of the time steps: providing a current sequence of audio data as input to a convolutional subnetwork, wherein the current sequence comprises the respective audio sample at each time step that precedes the time step in the output sequence, and wherein the convolutional subnetwork is configured to process the current sequence of audio data to generate an alternative representation for the time step; and providing the alternative representation for the time step as input to an output layer, wherein the output layer is configured to: process the alternative representation to generate an output that defines a score distribution over a plurality of possible audio samples for the time step.
1 . A method performed by one or more computers, the method comprising:
receiving data characterizing a sequence of text;
processing a network input comprising the data characterizing the sequence of text using a neural network to generate a neural network output that defines audio data that is a verbalization of the sequence of text characterized by the network input, wherein:
the neural network comprises a sequence of one or more convolutional neural network layers; and
each convolutional neural network layer in the sequence of one or more convolutional neural network layers is configured to:
receive a layer input;
apply one or more convolution operations to the layer input; and
generate a layer output based at least in part on a result of applying the one or more convolution operations to the layer input;
wherein the neural network output directly comprises amplitude values of an audio waveform that is the verbalization of the sequence of text characterized by the network input,
wherein the amplitude values of the audio waveform are directly generated as an output of an output neural network layer of the neural network, and
wherein the neural network has been trained by applying a supervised learning technique that depends on (i) ground truth output audio waveforms for each of a set of training examples for the neural network and (ii) corresponding output audio waveforms generated by the neural network.
2 . The method of claim 1 , wherein the data characterizing the sequence of text comprises a sequence of phonemes corresponding to the sequence of text.
3 . The method of claim 1 , wherein for each of one or more convolutional neural network layers in the sequence of one or more convolutional neural network layers, applying one or more convolution operations to the layer input comprises:
applying one or more dilated convolution operations to the layer input.
4 . The method of claim 3 , wherein the sequence of one or more convolutional neural network layers comprises a plurality of convolutional neural network layers that each implement respective dilated convolution operations associated with a respective different dilation rate.
5 . The method of claim 4 , wherein the sequence of convolutional neural network layers comprises a plurality of convolutional neural network layers that each have a dilation rate that is a constant multiple of a dilation rate of a preceding convolutional neural network layer.
6 . The method of claim 1 , wherein for each of one or more convolutional neural network layers in the sequence of one or more convolutional neural network layers, the convolutional neural network layer comprises a gated activation unit that is configured to:
process the layer input by a main convolutional operation to generate a main convolutional output;
process the layer input by a gate convolutional operation to generate a gating convolutional output; and
generate a gated activation unit output by element-wise multiplying the main convolutional output and the gating convolutional output.
7 . The method of claim 1 , wherein the sequence of one or more convolutional neural network layers comprises one or more residual connections, wherein each residual connection is configured to route the layer input to the convolutional neural network layer to a summer that sums the layer input with an intermediate output generated by the convolutional neural network layer, wherein the layer output is based at least in part on the sum of the layer input with the intermediate output.
8 . The method of claim 1 , wherein the sequence of one or more convolutional neural network layers comprises a plurality of convolutional neural network layers.
9 . The method of claim 1 , wherein the neural network is conditioned on speaker identity data; and
wherein the audio data that is the verbalization of the sequence of text is expressed in a voice associated with the speaker identity data.
10 . The method of claim 1 , further comprising:
evaluating an objective function that measures an error in the audio waveform; and
backpropagating gradients of the objective function through the neural network.
11 . The method of claim 1 , wherein:
the network input comprises data characterizing a conditioning input; and
wherein processing the network input comprising the data characterizing the sequence of text using the neural network to generate the neural network output that defines audio data that is the verbalization of the sequence of text characterized by the network input comprises:
processing the network input comprising the data characterizing the sequence of text using the neural network and the data characterizing the conditioning input to generate the neural network output that defines audio data that is the verbalization of the sequence of text characterized by the network input, as conditioned on the conditioning input.
12 . The method of claim 11 , wherein the conditioning input comprises image data.
13 . The method of claim 11 , wherein the conditioning input comprises video data.
14 . The method of claim 11 , wherein the conditioning input comprises data characterizing a particular speaker for the verbalization of the sequence of text.
15 . The method of claim 11 , wherein the conditioning input comprises data characterizing a particular language for the verbalization of the sequence of text.
16 . The method of claim 11 , wherein the conditioning input comprises data characterizing particular music for the verbalization of the sequence of text.
17 . A system comprising:
one or more computers; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
receiving data characterizing a sequence of text;
processing a network input comprising the data characterizing the sequence of text using a neural network to generate a neural network output that defines audio data that is a verbalization of the sequence of text characterized by the network input, wherein:
the neural network comprises a sequence of one or more convolutional neural network layers; and
each convolutional neural network layer in the sequence of one or more convolutional neural network layers is configured to:
receive a layer input;
apply one or more convolution operations to the layer input; and
generate a layer output based at least in part on a result of applying the one or more convolution operations to the layer input;
wherein the neural network output directly comprises amplitude values of an audio waveform that is the verbalization of the sequence of text characterized by the network input, and
wherein the amplitude values of the audio waveform are directly generated as an output of an output neural network layer of the neural network, and
wherein the neural network has been trained by applying a supervised learning technique that depends on (i) ground truth output audio waveforms for each of a set of training examples for the neural network and (ii) corresponding output audio waveforms generated by the neural network.
18 . The system of claim 17 , wherein the data characterizing the sequence of text comprises a sequence of phonemes corresponding to the sequence of text.
19 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
receiving data characterizing a sequence of text;
processing a network input comprising the data characterizing the sequence of text using a neural network to generate a neural network output that defines audio data that is a verbalization of the sequence of text characterized by the network input, wherein:
the neural network comprises a sequence of one or more convolutional neural network layers; and
each convolutional neural network layer in the sequence of one or more convolutional neural network layers is configured to:
receive a layer input;
apply one or more convolution operations to the layer input; and
generate a layer output based at least in part on a result of applying the one or more convolution operations to the layer input;
wherein the neural network output directly comprises amplitude values of an audio waveform that is the verbalization of the sequence of text characterized by the network input,
wherein the amplitude values of the audio waveform are directly generated as an output of an output neural network layer of the neural network, and
wherein the neural network has been trained by applying a supervised learning technique that depends on (i) ground truth output audio waveforms for each of a set of training examples for the neural network and (ii) corresponding output audio waveforms generated by the neural network.
20 . The non-transitory computer storage media of claim 19 , wherein the data characterizing the sequence of text comprises a sequence of phonemes corresponding to the sequence of text.