Granted Patent B2

US 12,620,408 · App. 18/519,986 · Granted May 5, 2026

Inventors: Aaron Gerard Antonius van den Oord (London, GB); Sander Etienne Lea Dieleman (London, GB); Nal Emmerich Kalchbrenner (Amsterdam, NL); Karen Simonyan (London, GB); Oriol Vinyals (London, GB)

Assignee: GDM Holding LLC

G10L25/30G06N3/045G06N3/0464G06N3/048G10L13/06G10H2250/311

Abstract

Claims (62)

1 . A method performed by one or more computers, the method comprising:

receiving data characterizing a sequence of text;

processing a network input comprising the data characterizing the sequence of text using a neural network to generate a neural network output that defines audio data that is a verbalization of the sequence of text characterized by the network input, wherein:

the neural network comprises a sequence of one or more convolutional neural network layers; and

each convolutional neural network layer in the sequence of one or more convolutional neural network layers is configured to:

receive a layer input;

apply one or more convolution operations to the layer input; and

generate a layer output based at least in part on a result of applying the one or more convolution operations to the layer input;

wherein the neural network output directly comprises amplitude values of an audio waveform that is the verbalization of the sequence of text characterized by the network input,

wherein the amplitude values of the audio waveform are directly generated as an output of an output neural network layer of the neural network, and

wherein the neural network has been trained by applying a supervised learning technique that depends on (i) ground truth output audio waveforms for each of a set of training examples for the neural network and (ii) corresponding output audio waveforms generated by the neural network.

2 . The method of claim 1 , wherein the data characterizing the sequence of text comprises a sequence of phonemes corresponding to the sequence of text.

3 . The method of claim 1 , wherein for each of one or more convolutional neural network layers in the sequence of one or more convolutional neural network layers, applying one or more convolution operations to the layer input comprises:

applying one or more dilated convolution operations to the layer input.

4 . The method of claim 3 , wherein the sequence of one or more convolutional neural network layers comprises a plurality of convolutional neural network layers that each implement respective dilated convolution operations associated with a respective different dilation rate.

5 . The method of claim 4 , wherein the sequence of convolutional neural network layers comprises a plurality of convolutional neural network layers that each have a dilation rate that is a constant multiple of a dilation rate of a preceding convolutional neural network layer.

6 . The method of claim 1 , wherein for each of one or more convolutional neural network layers in the sequence of one or more convolutional neural network layers, the convolutional neural network layer comprises a gated activation unit that is configured to:

process the layer input by a main convolutional operation to generate a main convolutional output;

process the layer input by a gate convolutional operation to generate a gating convolutional output; and

generate a gated activation unit output by element-wise multiplying the main convolutional output and the gating convolutional output.

7 . The method of claim 1 , wherein the sequence of one or more convolutional neural network layers comprises one or more residual connections, wherein each residual connection is configured to route the layer input to the convolutional neural network layer to a summer that sums the layer input with an intermediate output generated by the convolutional neural network layer, wherein the layer output is based at least in part on the sum of the layer input with the intermediate output.

8 . The method of claim 1 , wherein the sequence of one or more convolutional neural network layers comprises a plurality of convolutional neural network layers.

9 . The method of claim 1 , wherein the neural network is conditioned on speaker identity data; and

wherein the audio data that is the verbalization of the sequence of text is expressed in a voice associated with the speaker identity data.

10 . The method of claim 1 , further comprising:

evaluating an objective function that measures an error in the audio waveform; and

backpropagating gradients of the objective function through the neural network.

11 . The method of claim 1 , wherein:

the network input comprises data characterizing a conditioning input; and

wherein processing the network input comprising the data characterizing the sequence of text using the neural network to generate the neural network output that defines audio data that is the verbalization of the sequence of text characterized by the network input comprises:

processing the network input comprising the data characterizing the sequence of text using the neural network and the data characterizing the conditioning input to generate the neural network output that defines audio data that is the verbalization of the sequence of text characterized by the network input, as conditioned on the conditioning input.

12 . The method of claim 11 , wherein the conditioning input comprises image data.

13 . The method of claim 11 , wherein the conditioning input comprises video data.

14 . The method of claim 11 , wherein the conditioning input comprises data characterizing a particular speaker for the verbalization of the sequence of text.

15 . The method of claim 11 , wherein the conditioning input comprises data characterizing a particular language for the verbalization of the sequence of text.

16 . The method of claim 11 , wherein the conditioning input comprises data characterizing particular music for the verbalization of the sequence of text.

17 . A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

receiving data characterizing a sequence of text;

processing a network input comprising the data characterizing the sequence of text using a neural network to generate a neural network output that defines audio data that is a verbalization of the sequence of text characterized by the network input, wherein:

the neural network comprises a sequence of one or more convolutional neural network layers; and

each convolutional neural network layer in the sequence of one or more convolutional neural network layers is configured to:

receive a layer input;

apply one or more convolution operations to the layer input; and

generate a layer output based at least in part on a result of applying the one or more convolution operations to the layer input;

wherein the neural network output directly comprises amplitude values of an audio waveform that is the verbalization of the sequence of text characterized by the network input, and

wherein the amplitude values of the audio waveform are directly generated as an output of an output neural network layer of the neural network, and

wherein the neural network has been trained by applying a supervised learning technique that depends on (i) ground truth output audio waveforms for each of a set of training examples for the neural network and (ii) corresponding output audio waveforms generated by the neural network.

18 . The system of claim 17 , wherein the data characterizing the sequence of text comprises a sequence of phonemes corresponding to the sequence of text.

19 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving data characterizing a sequence of text;

processing a network input comprising the data characterizing the sequence of text using a neural network to generate a neural network output that defines audio data that is a verbalization of the sequence of text characterized by the network input, wherein:

the neural network comprises a sequence of one or more convolutional neural network layers; and

each convolutional neural network layer in the sequence of one or more convolutional neural network layers is configured to:

receive a layer input;

apply one or more convolution operations to the layer input; and

generate a layer output based at least in part on a result of applying the one or more convolution operations to the layer input;

wherein the neural network output directly comprises amplitude values of an audio waveform that is the verbalization of the sequence of text characterized by the network input,

wherein the amplitude values of the audio waveform are directly generated as an output of an output neural network layer of the neural network, and

wherein the neural network has been trained by applying a supervised learning technique that depends on (i) ground truth output audio waveforms for each of a set of training examples for the neural network and (ii) corresponding output audio waveforms generated by the neural network.

20 . The non-transitory computer storage media of claim 19 , wherein the data characterizing the sequence of text comprises a sequence of phonemes corresponding to the sequence of text.

Assignments (2)

Continuity (7)

References Cited (174)