Processing text sequences using neural networks

1. A language modeling system implemented by one or more computers, the language modeling system comprising:

a masked convolutional decoder neural network that comprises a plurality of masked convolutional neural network layers and is configured to generate a respective probability distribution over a set of possible target embeddings at each of a plurality of time steps, comprising, at each time step of the plurality of time steps:

processing target embeddings corresponding to previous time steps using the plurality of masked convolutional neural network layers of the masked convolutional decoder neural network to generate a current probability distribution over the set of possible target embeddings;

wherein each target embedding in the set of possible target embeddings corresponds to a respective character or word in a natural language; and

instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising using the respective probability distribution generated by the decoder neural network at each of the plurality of time steps to estimate a probability that a string represented by the target embeddings corresponding to the plurality of time steps belongs to the natural language.

2. The language modeling system of claim 1 , wherein one or more of the plurality of masked convolutional neural network layers are one-dimensional masked convolutional neural network layers.

3. The language modeling system of claim 1 , wherein one or more of the plurality of masked convolutional neural network layers are masked dilated convolutional neural network layers.

4. The language modeling system of claim 3 , wherein the plurality of masked convolutional neural network layers are grouped into a plurality of groups, and wherein, within each group, dilation rates of masked dilated convolutional neural network layers in the group are doubled every layer.

5. The language modeling system of claim 1 , wherein one or more of the plurality of masked convolutional neural network layers are wrapped in a residual block that contains one or more additional convolutional layers.

6. The language modeling system of claim 5 , wherein the residual block includes a rectified linear unit (ReLU) activation layer.

7. The language modeling system of claim 5 , wherein the residual block includes a multiplicative unit activation layer.

8. One or more non-transitory computer readable storage media storing instructions executable by a data processing apparatus and that upon such execution causes the data processing apparatus to perform language modeling operations comprising:

using a masked convolutional decoder neural network that comprises a plurality of masked convolutional neural network layers to generate a respective probability distribution over a set of possible target embeddings at each of a plurality of time steps, comprising, at each time step of the plurality of time steps:

processing target embeddings corresponding to previous time steps using the plurality of masked convolutional neural network layers of the masked convolutional decoder neural network to generate a current probability distribution over the set of possible target embeddings;

wherein each target embedding in the set of possible target embeddings corresponds to a respective character or word in a natural language; and

using the respective probability distribution generated by the decoder neural network at each of the plurality of time steps to estimate a probability that a string represented by the target embeddings corresponding to the plurality of time steps belongs to the natural language.

9. The non-transitory computer readable storage media of claim 8 , wherein one or more of the plurality of masked convolutional neural network layers are one-dimensional masked convolutional neural network layers.

10. The non-transitory computer readable storage media of claim 8 , wherein one or more of the plurality of masked convolutional neural network layers are masked dilated convolutional neural network layers.

11. The non-transitory computer readable storage media of claim 10 , wherein the plurality of masked convolutional neural network layers are grouped into a plurality of groups, and wherein, within each group, dilation rates of masked dilated convolutional neural network layers in the group are doubled every layer.

12. The non-transitory computer readable storage media of claim 8 , wherein one or more of the plurality of masked convolutional neural network layers are wrapped in a residual block that contains one or more additional convolutional layers.

13. The non-transitory computer readable storage media of claim 12 , wherein the residual block includes a rectified linear unit (ReLU) activation layer.

14. The non-transitory computer readable storage media of claim 12 , wherein the residual block includes a multiplicative unit activation layer.

15. A language modeling method performing by one or more data processing apparatus, the language modeling method comprising:

using a masked convolutional decoder neural network that comprises a plurality of masked convolutional neural network layers to generate a respective probability distribution over a set of possible target embeddings at each of a plurality of time steps, comprising, at each time step of the plurality of time steps:

processing target embeddings corresponding to previous time steps using the plurality of masked convolutional neural network layers of the masked convolutional decoder neural network to generate a current probability distribution over the set of possible target embeddings;

wherein each target embedding in the set of possible target embeddings corresponds to a respective character or word in a natural language; and

using the respective probability distribution generated by the decoder neural network at each of the plurality of time steps to estimate a probability that a string represented by the target embeddings corresponding to the plurality of time steps belongs to the natural language.

16. The language modeling method of claim 15 , wherein one or more of the plurality of masked convolutional neural network layers are one-dimensional masked convolutional neural network layers.

17. The language modeling method of claim 15 , wherein one or more of the plurality of masked convolutional neural network layers are masked dilated convolutional neural network layers.