Speech recognition using convolutional neural networks

1. An audio data processing neural network system implemented by one or more computers,

wherein the audio data processing neural network system is configured to perform an audio data sequence processing task by processing an audio data input sequence of audio data elements comprising a plurality of audio data elements to generate a neural network output which characterizes the audio data input sequence, and

wherein the audio data processing neural network system comprises:

a convolutional subnetwork comprising a plurality of dilated convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of audio data elements:

receive a current audio data input sequence comprising the audio data element and the audio data elements that precede the audio data element in the audio data input sequence, and

process the current audio data input sequence to generate a numeric representation for the audio data element, wherein the dilated convolutional neural network layers are causal convolutional neural network layers and the numeric representation for the audio data element does not depend on any audio data elements that follow the audio data element in the audio data input sequence; and

one or more output generation layers, wherein the one or more output generation layers are configured to receive the numeric representations of the audio data elements and to process the numeric representations of the audio data elements to generate the neural network output characterizing the audio data input sequence.

2. The audio data processing neural network system of claim 1 , wherein the dilated convolutional neural network layers include multiple blocks of dilated convolutional neural network layers, wherein each block comprises multiple dilated convolutional neural network layers with increasing dilation.

3. The audio data processing neural network system of claim 1 , wherein one or more of the dilated convolutional neural network layers have gated activation units.

4. The audio data processing neural network system of claim 1 , wherein the convolutional subnetwork comprises residual connections.

5. The audio data processing neural network system of claim 1 , wherein the convolutional subnetwork comprises skip connections.

6. The audio data processing neural network system of claim 1 ,

wherein the neural network output comprises a plurality of sets of scores,

wherein each set of scores includes a respective score for each of a plurality of possible outputs, and

wherein the audio data processing neural network system further comprises:

a subsystem configured to select outputs in accordance with the sets of scores to generate an output sequence comprising a plurality of outputs.

7. The audio data processing neural network system of claim 6 , wherein the audio data processing neural network system is configured to perform speech recognition, and the output sequence is a sequence of graphemes that is a grapheme representation of words spoken in the audio data input sequence.

8. The audio data processing neural network system of claim 6 , wherein the audio data processing neural network system is configured to perform speech recognition, and the output sequence is a sequence of phonemes that is a phoneme representation of words spoken in the audio data input sequence.

9. The audio data processing neural network system of claim 6 , wherein the audio data processing neural network system is configured to perform speech recognition, and the output sequence is a sequence of one or more words in a first natural language that represents worked spoken in the audio data input sequence.

10. The audio data processing neural network system of claim 6 , wherein the audio data processing neural network system is configured to perform a compression task, the audio data input sequence is a sequence of audio data elements drawn from an audio input sample, and the output sequence is a compressed or encoded sample.

11. The audio data processing neural network system of claim 6 , wherein the one or more output generation layers comprise:

a mean pooling layer configured to:

receive a subset of the numeric representations, and

perform mean pooling on the subset of numeric representations to generate a mean pooled representation for the subset; and

one or more additional neural network layers configured to, for each subset of the numeric representations, receive the mean pooled representation for the subset and to generate a set of scores for a position in the output sequence corresponding to the subset.

12. The audio data processing neural network system of claim 1 , wherein the audio data sequence processing task comprises speech recognition.

13. The audio data processing neural network system of claim 1 , wherein for one or more of the audio data elements that follow a first audio data element in the audio data input sequence, processing the current audio data input sequence to generate the numeric representation for the audio data element comprises:

retrieving stored values that were previously computed while generating respective numeric representations for one or more audio data elements that precede the audio data element in the input sequence; and

re-using the stored values while generating the numeric representation for the audio data element.

14. The audio data processing neural network system of claim 13 , wherein the stored values that were previously computed while generating respective numeric representations for one or more audio data elements that precede the audio data element in the input sequence comprise values derived from application of a convolutional filter to: (i) the audio data input sequence, or (ii) data derived from the audio data input sequence.

15. The audio data processing neural network system of claim 13 , wherein processing the current audio data input sequence to generate the numeric representation for the audio data element comprises:

storing one or more values computed while generating the numeric representation for the audio data element for use in generating respective numeric representations for audio data elements that follow the audio data element in the audio data input sequence.

16. A method of performing an audio data sequence processing task by generating a neural network output from an audio data input sequence of audio data elements comprising a plurality of audio data elements, wherein the neural network output characterizes the audio data input sequence, the method comprising:

for each of the plurality of audio data elements, providing a current audio data input sequence that comprises the audio data element and the audio data elements preceding the audio data element in the audio data input sequence to a convolutional subnetwork comprising a plurality of dilated convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of audio data elements:

receive the current audio data input sequence for the audio data element, and

process the current audio data input sequence to generate a numeric representation for the audio data element, wherein the dilated convolutional neural network layers are causal convolutional neural network layers and the numeric representation for the audio data element does not depend on any audio data elements that follow the audio data element in the audio data input sequence; and

providing the numeric representations of the audio data elements to one or more output generation layers that are configured to receive the numeric representations of the audio data elements and to process the numeric representations of the audio data elements to generate the neural network output characterizing the audio data input sequence.

17. The method of claim 16 , wherein the audio data sequence processing task comprises speech recognition.

18. The method of claim 16 , wherein the dilated convolutional neural network layers include multiple blocks of dilated convolutional neural network layers, wherein each block comprises multiple dilated convolutional neural network layers with increasing dilation.

19. The method of claim 16 , wherein one or more of the dilated convolutional neural network layers have gated activation units.

20. The method of claim 16 , wherein the convolutional subnetwork comprises residual connections.

21. The method of claim 16 , wherein for one or more of the audio data elements that follow a first audio data element in the audio data input sequence, processing the current audio data input sequence to generate the numeric representation for the audio data element comprises:

retrieving stored values that were previously computed while generating respective numeric representations for one or more audio data elements that precede the audio data element in the input sequence; and

re-using the stored values while generating the numeric representation for the audio data element.

22. The method of claim 21 , wherein the stored values that were previously computed while generating respective numeric representations for one or more audio data elements that precede the audio data element in the input sequence comprise values derived from application of a convolutional filter to: (i) the audio data input sequence, or (ii) data derived from the audio data input sequence.

23. The method of claim 21 , wherein processing the current audio data input sequence to generate the numeric representation for the audio data element further comprises:

storing one or more values computed while generating the numeric representation for the audio data element for use in generating respective numeric representations for audio data elements that follow the audio data element in the audio data input sequence.

24. One or more non-transitory computer storage media encoded with instructions that when executed by one or more computers cause the one or more computers to perform operations for accomplishing an audio data sequence processing task by generating a neural network output from an audio data input sequence of audio data elements comprising a plurality of audio data elements, wherein the neural network output characterizes the audio data input sequence, the operations comprising:

for each of the plurality of audio data elements, providing a current audio data input sequence that comprises the audio data element and the audio data elements preceding the audio data element in the audio data input sequence to a convolutional subnetwork comprising a plurality of dilated convolutional neural network layers, wherein the convolutional subnetwork is configured to, for each of the plurality of audio data elements:

receive the current audio data input sequence for the audio data element, and

process the current audio data input sequence to generate a numeric representation for the audio data element, wherein the dilated convolutional neural network layers are causal convolutional neural network layers and the numeric representation for the audio data element does not depend on any audio data elements that follow the audio data element in the audio data input sequence; and

providing the numeric representations of the audio data elements to one or more output generation layers that are configured to receive the numeric representations of the audio data elements and to process the numeric representations of the audio data elements to generate the neural network output characterizing the audio data input sequence.

25. The one or more non-transitory computer storage media of claim 24 , wherein the audio data sequence processing task comprises speech recognition.

26. The one or more non-transitory computer storage media of claim 24 , wherein the dilated convolutional neural network layers include multiple blocks of dilated convolutional neural network layers, wherein each block comprises multiple dilated convolutional neural network layers with increasing dilation.

27. The one or more non-transitory computer storage media of claim 24 , wherein one or more of the dilated convolutional neural network layers have gated activation units.

28. The one or more non-transitory computer storage media of claim 24 , wherein for one or more of the audio data elements that follow a first audio data element in the audio data input sequence, processing the current audio data input sequence to generate the numeric representation for the audio data element comprises:

retrieving stored values that were previously computed while generating respective numeric representations for one or more audio data elements that precede the audio data element in the input sequence; and

re-using the stored values while generating the numeric representation for the audio data element.

29. The one or more non-transitory computer storage media of claim 28 , wherein the stored values that were previously computed while generating respective numeric representations for one or more audio data elements that precede the audio data element in the input sequence comprise values derived from application of a convolutional filter to: (i) the audio data input sequence, or (ii) data derived from the audio data input sequence.

30. The one or more non-transitory computer storage media of claim 28 , wherein processing the current audio data input sequence to generate the numeric representation for the audio data element further comprises:

storing one or more values computed while generating the numeric representation for the audio data element for use in generating respective numeric representations for audio data elements that follow the audio data element in the audio data input sequence.