Generating images using neural networks

1. A neural network system implemented by one or more computers, the neural network system being configured to receive a neural network input and to generate an output image from the neural network input, the output image comprising a plurality of pixels arranged in a two-dimensional map, each pixel having a respective color value for each of a plurality of color channels, and the neural network system comprising:

one or more initial neural network layers configured to receive the neural network input and to process the neural network input to generate an alternative representation of the neural network input; and

one or more output layers, wherein the output layers are configured to receive the alternative representation and to generate the output image pixel by pixel from a sequence of pixels taken from the output image, comprising, for each pixel in the output image, generating a respective score distribution over a discrete set of possible color values for each of the plurality of color channels, wherein the plurality of color channels are ordered, and wherein the one or more output layers are configured to, for each color channel of each pixel of the output image;

generate the respective score distribution over the discrete set of possible color values for the color channel conditioned on (i) color values for pixels before the pixel in the sequence and (ii) color values for the pixel for any color channels before the color channel corresponding to the output layer in the order of color channels.

2. The neural network system of claim 1 , wherein, for each color channel of each pixel, each of the output layers is configured to receive a portion of the alternative representation corresponding to the color channel generated based on (i) color values for pixels before the pixel in the sequence and (ii) color values for the pixel for any color channels before the color channel in the order of color channels.

3. The neural network system of claim 2 , wherein the neural network system is configured to apply a mask to outputs of neural network layers in the one or more initial neural network layers such that the portion of the alternative representation corresponding to the color channel is only generated based on pixels before the pixel in the sequence and color channel data for the pixel for color channels before the color channel in the order of color channels.

4. The neural network system of claim 1 ,

wherein the neural network input is an image,

wherein the one or more initial neural network layers comprise a row long short-term memory (LSTM) layer, and wherein the row LSTM layer is configured to:

process an input image row by row from top to bottom computing features for a whole row at once.

5. The neural network system of claim 4 , wherein the row LSTM layer computes the features using a one-dimensional convolution.

6. The neural network system of claim 4 , wherein the row LSTM layer is configured to:

compute an input-to-state component of the row LSTM layer for the entire input image; and

after computing the input-to-state component for the entire input image, use the input-to-state component in processing the input image row by row from top to bottom computing features for a whole row at once.

7. The neural network system of claim 1 ,

wherein the neural network input is an image,

wherein the one or more initial neural network layers comprise a diagonal bidirectional LSTM (BiLSTM) layer, and wherein the diagonal BiLSTM layer is configured to:

scan an input image map in diagonal fashion along a first direction and in diagonal fashion along a second direction to generate features of the input image map.

8. The neural network system of claim 7 , wherein the diagonal BiLSTM layer is configured to:

skew the input image map into a space that allows for easily applying convolutions along diagonals; and

for each of the first direction and the second direction:

compute an input-to-state component of the diagonal biLSTM layer for the direction by applying a 1×1 convolution to the skewed input image map; and

compute a state-to-state recurrent component of the diagonal biLSTM layer for the direction by applying a column-wise convolution to the skewed input image map.

9. The neural network system of claim 1 , wherein the input is an image, and

wherein the one or more initial neural network layers comprise one or more convolutional neural network layers.

10. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement a neural network system configured to receive a neural network input and to generate an output image from the neural network input, the output image comprising a plurality of pixels arranged in a two-dimensional map, each pixel having a respective color value for each of a plurality of color channels, and the neural network system comprising:

one or more initial neural network layers configured to receive the neural network input and to process the neural network input to generate an alternative representation of the neural network input; and

one or more output layers, wherein the output layers are configured to receive the alternative representation and to generate the output image pixel by pixel from a sequence of pixels taken from the output image, comprising, for each pixel in the output image, generating a respective score distribution over a discrete set of possible color values for each of the plurality of color channels, wherein the plurality of color channels are ordered, and wherein the one or more output layers are configured to, for each color channel of each pixel of the output image:

generate the respective score distribution over the discrete set of possible color values for the color channel conditioned on (i) color values for pixels before the pixel in the sequence and (ii) color values for the pixel for any color channels before the color channel in the order of color channels.

11. The computer-readable storage media of claim 10 , wherein, for each color channel of each pixel, each of the output layers is configured to receive a portion of the alternative representation corresponding to the color channel generated based on (i) color values for pixels before the pixel in the sequence and (ii) color values for the pixel for any color channels before the color channel in the order of color channels.

12. The computer-readable storage media of claim 11 , wherein the neural network system is configured to apply a mask to outputs of neural network layers in the one or more initial neural network layers such that the portion of the alternative representation corresponding to the color channel is only generated based on pixels before the pixel in the sequence and color channel data for the pixel for color channels before the color channel in the order of color channels.

13. The computer-readable storage media of claim 10 ,

wherein the neural network input is an image,

wherein the one or more initial neural network layers comprise a row long short-term memory (LSTM) layer, and wherein the row LSTM layer is configured to:

process an input image row by row from top to bottom computing features for a whole row at once.

14. The computer-readable storage media of claim 13 , wherein the row LSTM layer computes the features using a one-dimensional convolution.

15. The computer-readable storage media of claim 14 , wherein the row LSTM layer is configured to:

compute an input-to-state component of the row LSTM layer for the entire input image; and

after computing the input-to-state component for the entire input image, use the input-to-state component in processing the input image row by row from top to bottom computing features for a whole row at once.

16. The computer-readable storage media of claim 10 ,

wherein the neural network input is an image,

wherein the one or more initial neural network layers comprise a diagonal bidirectional LSTM (BiLSTM) layer, and wherein the diagonal BiLSTM layer is configured to:

scan an input image map in diagonal fashion along a first direction and in diagonal fashion along a second direction to generate features of the input image map.

17. The computer-readable storage media of claim 16 , wherein the diagonal BiLSTM layer is configured to:

skew the input image map into a space that allows for easily applying convolutions along diagonals; and

for each of the first direction and the second direction:

compute an input-to-state component of the diagonal biLSTM layer for the direction by applying a 1×1 convolution to the skewed input image map; and

compute a state-to-state recurrent component of the diagonal biLSTM layer for the direction by applying a column-wise convolution to the skewed input image map.

18. The computer-readable storage media of claim 10 , wherein the input is an image, and

wherein the one or more initial neural network layers comprise one or more convolutional neural network layers.

19. A computer-implemented method of generating an output image from a neural network input, the output image comprising a plurality of pixels arranged in a two-dimensional map, each pixel having a respective color value for each of a plurality of color channels, and the method comprising:

processing the neural network input using one or more initial neural network layers configured to receive the neural network input and to process the neural network input to generate an alternative representation of the neural network input; and

processing the alternative representation using one or more output layers, wherein the output layers are configured to receive the alternative representation and to generate the output image pixel by pixel from a sequence of pixels taken from the output image, comprising, for each pixel in the output image, generating a respective score distribution over a discrete set of possible color values for each of the plurality of color channels, wherein the plurality of color channels are ordered, and wherein the one or more output layers are configured to, for each color channel of each pixel of the output image:

generate the respective score distribution over the discrete set of possible color values for the color channel conditioned on (i) color values for pixels before the pixel in the sequence and (ii) color values for the pixel for any color channels before the color channel in the order of color channels.

20. The method of claim 19 , wherein, for each color channel of each pixel, each of the output layers is configured to receive a portion of the alternative representation corresponding to the color channel generated based on (i) color values for pixels before the pixel in the sequence and (ii) color values for the pixel for any color channels before the color channel in the order of color channels.

21. The method of claim 20 , wherein processing the neural network input using one or more initial neural network layers comprises applying a mask to outputs of neural network layers in the one or more initial neural network layers such that the portion of the alternative representation corresponding to the color channel is only generated based on pixels before the pixel in the sequence and color channel data for the pixel for color channels before the color channel in the order of color channels.

22. The method of claim 19 ,

wherein the neural network input is an image,

wherein the one or more initial neural network layers comprise a row long short-term memory (LSTM) layer, and wherein the row LSTM layer is configured to:

process an input image row by row from top to bottom computing features for a whole row at once.

23. The method of claim 22 , wherein the row LSTM layer computes the features using a one-dimensional convolution.

24. The method of claim 22 , wherein the row LSTM layer is configured to:

compute an input-to-state component of the row LSTM layer for the entire input image; and

after computing the input-to-state component for the entire input image, use the input-to-state component in processing the input image row by row from top to bottom computing features for a whole row at once.

25. The method of claim 19 ,

wherein the neural network input is an image,

wherein the one or more initial neural network layers comprise a diagonal bidirectional LSTM (BiLSTM) layer, and wherein the diagonal BiLSTM layer is configured to:

scan an input image map in diagonal fashion along a first direction and in diagonal fashion along a second direction to generate features of the input image map.

26. The method of claim 25 , wherein the diagonal BiLSTM layer is configured to:

skew the input image map into a space that allows for easily applying convolutions along diagonals; and

for each of the first direction and the second direction:

compute an input-to-state component of the diagonal biLSTM layer for the direction by applying a 1×1 convolution to the skewed input image map; and

compute a state-to-state recurrent component of the diagonal biLSTM layer for the direction by applying a column-wise convolution to the skewed input image map.

27. The method of claim 19 , wherein the input is an image, and

wherein the one or more initial neural network layers comprise one or more convolutional neural network layers.