Feedforward generative neural networks
A feedforward generative neural network that generates an output example that includes multiple output samples of a particular type in a single neural network inference. Optionally, the generation may be conditioned on a context input. For example, the feedforward generative neural network may generate a speech waveform that is a verbalization of an input text segment conditioned on linguistic features of the text segment.
1 . A computer-implemented method of training a feedforward generative neural network for generating an audio waveform based on an input text segment using a single forward pass of the feedforward generative neural network,
wherein the training comprises:
obtaining a training context input specifying features of a text segment;
processing a training feedforward input comprising the training context input using a single forward pass of the feedforward generative neural network in accordance with current values of parameters of the feedforward generative neural network to generate a training feedforward output comprising output values for a plurality of generation time steps;
processing the training context input using an autoregressive generative neural network that has been trained to autoregressively generate a plurality of autoregressive outputs, wherein the autoregressive generative neural network is a different and separate neural network from the feedforward generative neural network, and the processing comprises, for each respective generation time step of the plurality of generation time steps, using a respective one of a plurality of forward passes of the autoregressive generative neural network to generate a respective autoregressive output for the respective generation time step conditioned on (i) the training context input and (ii) output samples defined by the training feedforward output at one or more generation time steps that precede the respective generation time step;
determining a first gradient with respect to the parameters of the feedforward generative neural network to minimize a divergence loss that depends on a plurality of first divergences that include, for each of the plurality of generation time steps, a respective first divergence from a distribution defined by the autoregressive output and a distribution for the generation time step defined by the training feedforward output; and
determining an update to the current values of the parameters of the feedforward generative neural network based at least in part on the first gradient.
2 . The method of claim 1 , wherein the training feedforward input further comprises a respective noise input at each of the generation time steps.
3 . The method of claim 1 , wherein the training further comprises:
obtaining a ground-truth output example for the training context input; and
generating, from the training feedforward output, a predicted output example by sampling from the distributions defined by the training feedforward output.
4 . The method of claim 3 , wherein the ground-truth output example and the predicted output example are speech waveforms, wherein the training further comprises:
generating a first magnitude spectrogram of the ground-truth output example;
generating a second magnitude spectrogram of the predicted output example; and
determining a second gradient with respect to the feedforward parameters to minimize a magnitude loss that depends on a difference between the first and second magnitude spectrograms, and wherein determining the update to the current values of the feedforward parameters comprises determining the update based at least in part on the second gradient.
5 . The method of claim 3 , wherein the training further comprises:
processing the ground-truth output example using a trained feature generation neural network to obtain features of the ground-truth output example, wherein the trained feature generation neural network is a pre-trained neural network that takes a waveform as input;
processing the predicted output example using the trained feature generation neural network to obtain features of the predicted output example; and
determining a third gradient with respect to the feedforward parameters to minimize a perceptual loss that depends on a measure of difference between the features of the ground-truth output example and the features of the predicted output example, and wherein determining the update to the current values of the feedforward parameters comprises determining the update based at least in part on the third gradient.
6 . The method of claim 5 , wherein the feature generation neural network is a speech recognition neural network.
7 . The method of claim 5 , wherein the features are outputs of an intermediate layer in the feature generation network.
8 . The method of claim 5 , wherein the feature generation neural network is a trained autoregressive generative neural network.
9 . The method of claim 1 , wherein the training further comprises:
obtaining a different context input;
processing the different context input using the trained autoregressive generative neural network to obtain, for each of the plurality of generation time steps, a respective different autoregressive output; and
determining a fourth gradient with respect to the feedforward parameters to maximize a contrastive loss that depends at least in part on, for each of the generation time steps, a second divergence from a likelihood distribution defined by the different autoregressive output for the generation time step and the likelihood distribution for the generation time step defined by the training feedforward output, and wherein determining the update to the current values of the feedforward parameters comprises determining the update based at least in part on the fourth gradient.
10 . The method of claim 1 , wherein the first divergence is a KL divergence.
11 . The method of claim 1 , wherein the first divergence is a Jensen -Shannon Divergence.
12 . The method of claim 1 , wherein the divergence loss depends at least in part on a sum of the first divergences at each of the generation time steps.
13 . A method of generating an output audio waveform for an input text segment, the method comprising:
receiving a context input including features of the input text segment; and
generating the output audio waveform by processing a feedforward network input comprising the context input using a feedforward generative neural network using a single forward pass of the feedforward generative neural network, a feedforward output that defines an output example of the output audio waveform, wherein the output example comprises a plurality of output samples that include, for each of a plurality of generation time steps, a respective output sample at the respective generation time step, and
wherein the feedforward generative neural network has been trained using a training procedure comprising:
obtaining a training context input specifying features of a text segment;
processing a training feedforward input comprising the training context input using a single forward pass of the feedforward generative neural network in accordance with current values of parameters of the feedforward generative neural network to generate a training feedforward output comprising output values for a plurality of generation time steps;
processing the training context input using an autoregressive generative neural network that has been trained to autoregressively generate a plurality of autoregressive outputs, wherein the autoregressive generative neural network is a different and separate neural network from the feedforward generative neural network, and the processing comprises, for each respective generation time step of the plurality of generation time steps, using a respective one of a plurality of forward passes of the autoregressive generative neural network to generate a respective autoregressive output for the respective generation time step conditioned on (i) the training context input and (ii) output samples defined by the training feedforward output that has been generated for one or more generation time steps that precede the respective generation time step;
determining a first gradient with respect to the parameters of the feedforward generative neural network to minimize a divergence loss that depends on a plurality of first divergences that include, for each of the plurality of generation time steps, a respective first divergence from a likelihood distribution defined by the autoregressive output and a likelihood distribution for the generation time step defined by the training feedforward output; and
determining an update to the current values of the parameters of the feedforward generative neural network based at least in part on the first gradient.
14 . The method of claim 13 , wherein the feedforward network input further comprises a noise input.
15 . A system comprising:
one or more computers; and
one or more storage devices storing instructions that when executed by the one or more computers, cause the one or more computers to perform operations for training a feedforward generative neural network for generating an audio waveform based on an input text segment using a single forward pass of the feedforward generative neural network,
wherein the training comprises:
obtaining a training context input specifying features of a text segment;
processing a training feedforward input comprising the training context input using a single forward pass of the feedforward generative neural network in accordance with current values of parameters of the feedforward generative neural network to generate a training feedforward output comprising output values for a plurality of generation time steps;
processing the training context input using an autoregressive generative neural network that has been trained to autoregressively generate a plurality of autoregressive outputs, wherein the autoregressive generative neural network is a different and separate neural network from the feedforward generative neural network, and the processing comprises, for each respective generation time step of the plurality of generation time steps, using a respective one of a plurality of forward passes of the autoregressive generative neural network to generate a respective autoregressive output for the respective generation time step conditioned on (i) the training context input and (ii) output samples defined by the training feedforward output at one or more generation time steps that precede the respective generation time step;
determining a first gradient with respect to the parameters of the feedforward generative neural network to minimize a divergence loss that depends on a plurality of first divergences that include, for each of the plurality of generation time steps, a respective first divergence from a distribution defined by the autoregressive output and a distribution for the generation time step defined by the training feedforward output; and
determining an update to the current values of the parameters of the feedforward generative neural network based at least in part on the first gradient.
16 . The system of claim 15 , wherein the training feedforward input further comprises a respective noise input at each of the generation time steps.
17 . The system of claim 15 , wherein the training further comprises:
obtaining a ground-truth output example for the training context input; and
generating, from the training feedforward output, a predicted output example by sampling from the distributions defined by the training feedforward output.
18 . The system of claim 17 , wherein the ground-truth output example and the predicted output example are speech waveforms, wherein the training further comprises:
generating a first magnitude spectrogram of the ground-truth output example;
generating a second magnitude spectrogram of the predicted output example; and
determining a second gradient with respect to the parameters of the feedforward generative neural network to minimize a magnitude loss that depends on a difference between the first and second magnitude spectrograms, and wherein determining the update to the current values of the parameters comprises determining the update based at least in part on the second gradient.
19 . The system of claim 17 , wherein the training further comprises:
processing the ground-truth output example using a trained feature generation neural network to obtain features of the ground-truth output example, wherein the trained feature generation neural network is a pre-trained neural network that takes a waveform as input;
processing the predicted output example using the trained feature generation neural network to obtain features of the predicted output example; and
determining a third gradient with respect to the parameters to minimize a perceptual loss that depends on a measure of difference between the features of the ground-truth output example and the features of the predicted output example, and wherein determining the update to the current values of the parameters comprises determining the update based at least in part on the third gradient.