Augmented recurrent neural network with external memory

1. An augmented neural network system for processing one or more inputs to generate a sequence of system outputs, the augmented neural network system comprising:

a recurrent neural network, wherein the recurrent neural network is configured to receive a network input at each of a plurality of time steps and to process the network input and a current internal state of the recurrent neural network to generate a neural network output for the time step and to update the current internal state of the recurrent neural network; and

a memory interface subsystem, wherein the memory interface subsystem is configured to perform operations comprising, for each of the plurality of time steps:

providing an output derived from the neural network output for the time step as a system output for the time step;

determining, from the neural network output for the time step, memory state parameters for the time step;

updating a current state of an external memory using the memory state parameters for the time step, wherein the external memory is configured to store data vectors written to the external memory by the memory interface subsystem, wherein the external memory assigns each data vector to a respective row in a matrix memory, wherein the external memory is further configured to store a size vector having a respective entry for each row in the matrix memory, and wherein updating the current state comprises:

writing, based on the memory state parameters determined from the neural network output for the time step, data to one or more new rows of the matrix memory; and

updating, based on the memory state parameters determined from the neural network output for the time step, the size vector, comprising:

adding a respective new entry to the size vector for each new row of the matrix memory;

reading data from the external memory in accordance with the updated state of the external memory, wherein reading data from the external memory comprises:

generating, from the entries in the updated size vector, a respective weight for each row of the matrix memory, and

reading a read vector from the external memory by computing a weighted sum of the rows of the matrix memory in accordance with the respective weights; and

combining the data read from the external memory with a system input for the next time step to generate the neural network input for the next time step.

2. The augmented neural network system of claim 1 , wherein determining, from the neural network output for the time step, memory state parameters comprises:

determining, from the neural network output for the time step, a pop value for the time step;

determining, from the neural network output for the time step, a push value for the time step; and

determining, from the neural network output for the time step, a write vector for the time step.

3. The augmented neural network system of claim 2 , wherein updating the current state of the external memory comprises:

adding the write vector as a new row of the matrix.

4. The augmented neural network system of claim 3 , wherein updating the size vector comprises:

updating the entries of the size vector using the pop value to update the weights for the vectors in the rows of the matrix; and

adding the push value to the size vector as an entry corresponding to the new row of the matrix.

5. The augmented neural network system of claim 4 , wherein the respective weights are the corresponding entries of the size vector.

6. The augmented neural network system of claim 5 , wherein combining the data read from the external memory with the system input for the next time step to generate the neural network input for the next time step comprises:

concatenating the read vector and the system input to generate the neural network input for the next time step.

7. The augmented neural network system of claim 1 , wherein determining, from the neural network output for the time step, memory state parameters comprises:

determining, from the neural network output for the time step, a top pop value for the time step and a bottom pop value for the time step;

determining, from the neural network output for the time step, a top push value for the time step and a bottom push value for the time step; and

determining, from the neural network output for the time step, a top write vector for the time step and a bottom write vector for the time step.

8. The augmented neural network system of claim 7 , wherein updating the current state of the external memory comprises:

adding the top write vector as a new top row of the matrix; and

adding the bottom write vector as a new bottom row of the matrix.

9. The augmented neural network system of claim 8 , wherein updating the current state of the external memory comprises:

updating each current entry of the size vector based on the current entry and the current entries after the current entry in the size vector using the top pop value to generate initial updated entries of the size vector;

updating each initial updated entry of the size vector based on the initial updated entry and the initial updated entries before the initial updated entry in the size vector using the bottom pop value to generate final updated entries; and

adding the top push value to the size vector as an entry corresponding to the new top row of the matrix and the bottom push value to the size vector as an entry corresponding to the new bottom row of the matrix.

10. The augmented neural network system of claim 9 , wherein reading data from the external memory in accordance with the updated internal state comprises:

reading a top read vector from the external memory, wherein each entry of the top read vector is a weighted sum of the rows of the matrix, and wherein each row is weighted according to the corresponding entry in the size vector and the entries after the corresponding entry in the size vector; and

reading a bottom read vector from the external memory, wherein each entry of the top read vector is a weighted sum of the rows of the matrix, and wherein each row is weighted according to the corresponding entry in the size vector and the entries before the corresponding entry in the size vector.

11. The augmented neural network system of claim 10 , wherein combining the data read from the external memory with the system input for the next time step to generate the neural network input for the next time step comprises:

concatenating the top read vector, the bottom read vector, and the system input to generate the neural network input for the next time step.

12. A method for processing one or more inputs to generate a sequence of system outputs using an augmented neural network system comprising a recurrent neural network,

wherein the recurrent neural network is configured to receive a network input at each of a plurality of time steps and to process the network input and a current internal state of the recurrent neural network to generate a neural network output for the time step and to update the current internal state of the recurrent neural network, and

wherein the method comprises, for each of the plurality of time steps:

providing an output derived from the neural network output for the time step as a system output for the time step;

determining, from the neural network output for the time step, memory state parameters for the time step;

updating a current state of an external memory using the memory state parameters for the time step, wherein the external memory is configured to store data vectors written to the external memory, wherein the external memory assigns each data vector to a respective row in a matrix memory, wherein the external memory is further configured to store a size vector having a respective entry for each row in the matrix memory, and wherein updating the current state comprises:

writing, based on the memory state parameters determined from the neural network output for the time step, data to one or more new rows of the matrix memory; and

updating, based on the memory state parameters determined from the neural network output for the time step, the size vector, comprising:

adding a respective new entry to the size vector for each new row of the matrix memory;

reading data from the external memory in accordance with the updated state of the external memory, wherein reading data from the external memory comprises:

generating, from the entries in the updated size vector, a respective weight for each row of the matrix memory, and

reading a read vector from the external memory by computing a weighted sum of the rows of the matrix memory in accordance with the respective weights; and

combining the data read from the external memory with a system input for the next time step to generate the neural network input for the next time step.

13. The method of claim 12 , wherein determining, from the neural network output for the time step, memory state parameters comprises:

determining, from the neural network output for the time step, a pop value for the time step;

determining, from the neural network output for the time step, a push value for the time step; and

determining, from the neural network output for the time step, a write vector for the time step.

14. The method of claim 13 , wherein updating the current state of the external memory comprises:

adding the write vector as a new row of the matrix.

15. The method of claim 14 , wherein updating the size vector comprises:

updating the entries of the size vector using the pop value to update the weights for the vectors in the rows of the matrix; and

adding the push value to the size vector as an entry corresponding to the new row of the matrix.

16. The method of claim 15 , wherein the respective weights are the corresponding entries of the size vector.

17. The method of claim 16 , wherein combining the data read from the external memory with the system input for the next time step to generate the neural network input for the next time step comprises:

concatenating the read vector and the system input to generate the neural network input for the next time step.

18. The method of claim 12 , wherein determining, from the neural network output for the time step, memory state parameters comprises:

determining, from the neural network output for the time step, a top pop value for the time step and a bottom pop value for the time step;

determining, from the neural network output for the time step, a top push value for the time step and a bottom push value for the time step; and

determining, from the neural network output for the time step, a top write vector for the time step and a bottom write vector for the time step.

19. The method of claim 18 , wherein updating the current state of the external memory comprises:

adding the top write vector as a new top row of the matrix; and

adding the bottom write vector as a new bottom row of the matrix.

20. The method of claim 19 , wherein updating the current state of the external memory comprises:

updating each current entry of the size vector based on the current entry and the current entries after the current entry in the size vector using the top pop value to generate initial updated entries of the size vector;

updating each initial updated entry of the size vector based on the initial updated entry and the initial updated entries before the initial updated entry in the size vector using the bottom pop value to generate final updated entries; and

adding the top push value to the size vector as an entry corresponding to the new top row of the matrix and the bottom push value to the size vector as an entry corresponding to the new bottom row of the matrix.

21. The method of claim 20 , wherein reading data from the external memory in accordance with the updated internal state comprises:

reading a top read vector from the external memory, wherein each entry of the top read vector is a weighted sum of the rows of the matrix, and wherein each row is weighted according to the corresponding entry in the size vector and the entries after the corresponding entry in the size vector; and

reading a bottom read vector from the external memory, wherein each entry of the top read vector is a weighted sum of the rows of the matrix, and wherein each row is weighted according to the corresponding entry in the size vector and the entries before the corresponding entry in the size vector.

22. The method of claim 21 , wherein combining the data read from the external memory with the system input for the next time step to generate the neural network input for the next time step comprises:

concatenating the top read vector, the bottom read vector, and the system input to generate the neural network input for the next time step.

23. One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to implement an augmented neural network system for processing one or more inputs to generate a sequence of system outputs, the augmented neural network system comprising:

a recurrent neural network, wherein the recurrent neural network is configured to receive a network input at each of a plurality of time steps and to process the network input and a current internal state of the recurrent neural network to generate a neural network output for the time step and to update the current internal state of the recurrent neural network; and

a memory interface subsystem, wherein the memory interface subsystem is configured to perform operations comprising, for each of the plurality of time steps:

providing an output derived from the neural network output for the time step as a system output for the time step;

determining, from the neural network output for the time step, memory state parameters for the time step;

updating a current state of an external memory using the memory state parameters for the time step, wherein the external memory is configured to store data vectors written to the external memory by a memory interface subsystem, wherein the external memory assigns each data vector to a respective row in a matrix memory, wherein the external memory is further configured to store a size vector having a respective entry for each row in the matrix memory, and wherein updating the current state comprises:

writing, based on the memory state parameters determined from the neural network output for the time step, data to one or more new rows of the matrix memory; and

updating, based on the memory state parameters determined from the neural network output for the time step, the size vector, comprising:

adding a respective new entry to the size vector for each new row of the matrix memory;

reading data from the external memory in accordance with the updated state of the external memory, wherein reading data from the external memory comprises:

generating, from the entries in the updated size vector, a respective weight for each row of the matrix memory, and

reading a read vector from the external memory by computing a weighted sum of the rows of the matrix memory in accordance with the respective weights; and

combining the data read from the external memory with a system input for the next time step to generate the neural network input for the next time step.