Grid long short-term memory neural networks
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing grid Long Short-Term Memory (LSTM) neural networks that includes a plurality of N-LSTM blocks arranged in an N-dimensional grid. Each N-LSTM block is configured to: receive N input hidden vectors, the N input hidden vectors each corresponding to a respective one of the N dimensions; receive N input memory vectors, the N input memory vectors each corresponding to a respective one of the N dimensions; and, for each of the dimensions, apply a respective transform for the dimension to the memory hidden vector corresponding to the dimension and the input hidden vector corresponding to the dimension to generate a new hidden vector corresponding to the dimension and a new memory vector corresponding to the dimension.
1. A system for processing a neural network input to generate a neural network output, the system comprising:
a grid Long Short-Term Memory (LSTM) neural network implemented by one or more computers, wherein the grid LSTM neural network comprises a plurality of N-LSTM blocks arranged in an N-dimensional grid, wherein N is an integer greater than zero, and wherein each N-LSTM block is configured to:
receive N input hidden vectors, the N input hidden vectors each corresponding to a respective one of the N dimensions;
receive N input memory vectors, the N input memory vectors each corresponding to a respective one of the N dimensions; and
for each of the dimensions, apply a respective transform for the dimension to the memory hidden vector corresponding to the dimension and the input hidden vector corresponding to the dimension to generate a new hidden vector corresponding to the dimension and a new memory vector corresponding to the dimension.
2. The system of claim 1 , wherein for each of the dimensions, applying a respective transform comprises:
generating a concatenated hidden vector for the dimension; and
applying the respective transform for the dimension to the concatenated hidden vector for the dimension.
3. The system of claim 2 , wherein, for one or more of the dimensions, the concatenated hidden vector is a concatenation of the N input hidden vectors.
4. The system of claim 2 , wherein, for one or more of the dimensions, the concatenated hidden vector is a concatenation of the input hidden vector for the dimension and the new hidden vectors for each other dimension.
5. The system of claim 2 , wherein, for one or more of the dimensions, applying the respective transform comprises applying a LSTM transform.
6. The system of claim 2 , wherein, for one or more of the dimensions, applying the respective transform comprises applying a non-LSTM transform.
7. The system of claim 1 , wherein, for a first N-LSTM block of the plurality of N-LSTM blocks, the input hidden vector and the input memory vector corresponding to a particular dimension are at least a portion of the neural network input.
8. The system of claim 7 , wherein the grid LSTM neural network further comprises an output layer configured to:
process the new hidden vector and the new memory vector corresponding to the particular dimension and generated by a second N-LSTM block of the plurality of N-LSTM blocks to generate the neural network output.
9. The system of claim 1 , wherein N is 1.
10. The system of claim 1 , wherein N is 2.
11. The system of claim 1 , wherein N is 3.
12. The system of claim 1 , wherein N is greater than 1.
13. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to implement:
a grid Long Short-Term Memory (LSTM) neural network, wherein the grid LSTM neural network comprises a plurality of N-LSTM blocks arranged in an N-dimensional grid, wherein N is an integer greater than zero, and wherein each N-LSTM block is configured to:
receive N input hidden vectors, the N input hidden vectors each corresponding to a respective one of the N dimensions;
receive N input memory vectors, the N input memory vectors each corresponding to a respective one of the N dimensions; and
for each of the dimensions, apply a respective transform for the dimension to the memory hidden vector corresponding to the dimension and the input hidden vector corresponding to the dimension to generate a new hidden vector corresponding to the dimension and a new memory vector corresponding to the dimension.
14. The one or more non-transitory computer storage media of claim 13 , wherein for each of the dimensions, applying a respective transform comprises:
generating a concatenated hidden vector for the dimension; and
applying the respective transform for the dimension to the concatenated hidden vector for the dimension.
15. The one or more non-transitory computer storage media of claim 14 , wherein, for one or more of the dimensions, the concatenated hidden vector is a concatenation of the N input hidden vectors.
16. The one or more non-transitory computer storage media of claim 14 , wherein, for one or more of the dimensions, the concatenated hidden vector is a concatenation of the input hidden vector for the dimension and the new hidden vectors for each other dimension.
17. The one or more non-transitory computer storage media of claim 14 , wherein, for one or more of the dimensions, applying the respective transform comprises applying a LSTM transform.
18. The one or more non-transitory computer storage media of claim 13 , wherein, for a first N-LSTM block of the plurality of N-LSTM blocks, the input hidden vector and the input memory vector corresponding to a particular dimension are at least a portion of a neural network input.
19. The one or more non-transitory computer storage media of claim 18 , wherein the grid LSTM neural network further comprises an output layer configured to:
process the new hidden vector and the new memory vector corresponding to the particular dimension and generated by a second N-LSTM block of the plurality of N-LSTM blocks to generate a neural network output.
20. A method for processing a neural network input to generate a neural network output by processing the neural network input using a grid Long Short-Term Memory (LSTM) neural network, wherein the grid LSTM neural network comprises a plurality of N-LSTM blocks arranged in an N-dimensional grid, wherein N is an integer greater than zero, and wherein the method comprises, for each N-LSTM block:
receiving N input hidden vectors, the N input hidden vectors each corresponding to a respective one of the N dimensions;
receiving N input memory vectors, the N input memory vectors each corresponding to a respective one of the N dimensions; and
for each of the dimensions, applying a respective transform for the dimension to the memory hidden vector corresponding to the dimension and the input hidden vector corresponding to the dimension to generate a new hidden vector corresponding to the dimension and a new memory vector corresponding to the dimension.