Augmented recurrent neural network with external memory

1. An augmented neural network system for processing one or more inputs to generate one or more system outputs, the augmented neural network system comprising:

a neural network, wherein the neural network is configured to receive a first network input and to process the first network input to generate a neural network output; and

a memory interface subsystem, wherein the memory interface subsystem is configured to perform operations comprising:

providing an output derived from the neural network output as a system output;

determining, from the neural network output, memory state parameters;

updating a current state of an external memory using the memory state parameters, wherein the external memory is configured to store data vectors written to the external memory by the memory interface subsystem, wherein the external memory assigns each data vector to a respective row in a matrix memory, wherein the external memory is further configured to store a size vector having a respective entry for each row in the matrix memory, and wherein updating the current state comprises:

writing, based on the memory state parameters determined from the neural network output, data to one or more new rows of the matrix memory; and

updating, based on the memory state parameters determined from the neural network output, the size vector, comprising:

adding a respective new entry to the size vector for each new row of the matrix memory;

reading data from the external memory in accordance with the updated state of the external memory, wherein reading data from the external memory comprises:

generating, from the entries in the updated size vector, a respective weight for each row of the matrix memory, and

reading a read vector from the external memory by computing a weighted sum of the rows of the matrix memory in accordance with the respective weights; and

combining the data read from the external memory with a system input to generate a second neural network input.

2. The augmented neural network system of claim 1 , wherein determining, from the neural network output, memory state parameters comprises:

determining, from the neural network output, a pop value;

determining, from the neural network output, a push value; and

determining, from the neural network output, a write vector.

3. The augmented neural network system of claim 2 , wherein updating the current state of the external memory comprises:

adding the write vector as a new row of the matrix.

4. The augmented neural network system of claim 3 , wherein updating the size vector comprises:

updating the entries of the size vector using the pop value to update the weights for the vectors in the rows of the matrix; and

adding the push value to the size vector as an entry corresponding to the new row of the matrix.

5. The augmented neural network system of claim 4 , wherein the respective weights are the corresponding entries of the size vector.

6. The augmented neural network system of claim 5 , wherein combining the data read from the external memory with the system input to generate the second neural network input comprises:

concatenating the read vector and the system input to generate the second neural network input.

7. The augmented neural network system of claim 1 , wherein determining, from the neural network output, memory state parameters comprises:

determining, from the neural network output, a top pop value and a bottom pop value;

determining, from the neural network output, a top push value and a bottom push value; and

determining, from the neural network output, a top write vector and a bottom write vector.

8. The augmented neural network system of claim 7 , wherein updating the current state of the external memory comprises:

adding the top write vector as a new top row of the matrix; and

adding the bottom write vector as a new bottom row of the matrix.

9. The augmented neural network system of claim 8 , wherein updating the current state of the external memory comprises:

updating each current entry of the size vector based on the current entry and the current entries after the current entry in the size vector using the top pop value to generate initial updated entries of the size vector;

updating each initial updated entry of the size vector based on the initial updated entry and the initial updated entries before the initial updated entry in the size vector using the bottom pop value to generate final updated entries; and

adding the top push value to the size vector as an entry corresponding to the new top row of the matrix and the bottom push value to the size vector as an entry corresponding to the new bottom row of the matrix.

10. The augmented neural network system of claim 9 , wherein reading data from the external memory in accordance with the updated internal state comprises:

reading a top read vector from the external memory, wherein each entry of the top read vector is a weighted sum of the rows of the matrix, and wherein each row is weighted according to the corresponding entry in the size vector and the entries after the corresponding entry in the size vector; and

reading a bottom read vector from the external memory, wherein each entry of the top read vector is a weighted sum of the rows of the matrix, and wherein each row is weighted according to the corresponding entry in the size vector and the entries before the corresponding entry in the size vector.

11. The augmented neural network system of claim 10 , wherein combining the data read from the external memory with the system input to generate the second neural network input comprises:

concatenating the top read vector, the bottom read vector, and the system input to generate the second neural network input.

12. A method for processing one or more inputs to generate a sequence of system outputs using an augmented neural network system comprising a neural network,

wherein the neural network is configured to receive a first network input and to process the first network input to generate a neural network output, and

wherein the method comprises:

providing an output derived from the neural network output as a system output;

determining, from the neural network output, memory state parameters;

updating a current state of an external memory using the memory state parameters, wherein the external memory is configured to store data vectors written to the external memory, wherein the external memory assigns each data vector to a respective row in a matrix memory, wherein the external memory is further configured to store a size vector having a respective entry for each row in the matrix memory, and wherein updating the current state comprises:

writing, based on the memory state parameters determined from the neural network output, data to one or more new rows of the matrix memory; and

updating, based on the memory state parameters determined from the neural network output, the size vector, comprising:

adding a respective new entry to the size vector for each new row of the matrix memory;

reading data from the external memory in accordance with the updated state of the external memory, wherein reading data from the external memory comprises:

generating, from the entries in the updated size vector, a respective weight for each row of the matrix memory, and

reading a read vector from the external memory by computing a weighted sum of the rows of the matrix memory in accordance with the respective weights; and

combining the data read from the external memory with a system input to generate a second neural network input.

13. The method of claim 12 , wherein determining, from the neural network output, memory state parameters comprises:

determining, from the neural network output, a pop value;

determining, from the neural network output, a push value; and

determining, from the neural network output, a write vector.

14. The method of claim 13 , wherein updating the current state of the external memory comprises:

adding the write vector as a new row of the matrix.

15. The method of claim 14 , wherein updating the size vector comprises:

updating the entries of the size vector using the pop value to update the weights for the vectors in the rows of the matrix; and

adding the push value to the size vector as an entry corresponding to the new row of the matrix.

16. The method of claim 15 , wherein the respective weights are the corresponding entries of the size vector.

17. The method of claim 16 , wherein combining the data read from the external memory with the system input to generate the second neural network input comprises:

concatenating the read vector and the system input to generate the second neural network input.

18. The method of claim 12 , wherein determining, from the neural network output, memory state parameters comprises:

determining, from the neural network output, a top pop value and a bottom pop value;

determining, from the neural network output, a top push value and a bottom push value; and

determining, from the neural network output, a top write vector and a bottom write vector.

19. The method of claim 18 , wherein updating the current state of the external memory comprises:

adding the top write vector as a new top row of the matrix; and

adding the bottom write vector as a new bottom row of the matrix.

20. One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to implement an augmented neural network system for processing one or more inputs to generate one or more system outputs, the augmented neural network system comprising:

a neural network, wherein the neural network is configured to receive a first network input and to process the first network input to generate a neural network output; and

a memory interface subsystem, wherein the memory interface subsystem is configured to perform operations comprising:

providing an output derived from the neural network output as a system output;

determining, from the neural network output, memory state parameters;

updating a current state of an external memory using the memory state parameters, wherein the external memory is configured to store data vectors written to the external memory by the memory interface subsystem, wherein the external memory assigns each data vector to a respective row in a matrix memory, wherein the external memory is further configured to store a size vector having a respective entry for each row in the matrix memory, and wherein updating the current state comprises:

writing, based on the memory state parameters determined from the neural network output, data to one or more new rows of the matrix memory; and

updating, based on the memory state parameters determined from the neural network output, the size vector, comprising:

adding a respective new entry to the size vector for each new row of the matrix memory;

reading data from the external memory in accordance with the updated state of the external memory, wherein reading data from the external memory comprises:

generating, from the entries in the updated size vector, a respective weight for each row of the matrix memory, and

reading a read vector from the external memory by computing a weighted sum of the rows of the matrix memory in accordance with the respective weights; and

combining the data read from the external memory with a system input to generate a second neural network input.