Augmenting neural networks with external memory

1. A method for processing a sequence of system inputs to generate a sequence of system outputs using an augmented neural network system comprising a neural network and an external memory, wherein the neural network is configured to receive a sequence of neural network inputs and to process each neural network input to generate a neural network output from the neural network input, wherein the external memory is external to the neural network and is configured to store a respective value vector in each of a plurality of locations in the external memory, and wherein the method comprises, for each neural network output:

providing an output derived from a first portion of the neural network output as a system output in the sequence of system outputs;

determining one or more sets of writing weights for each of the plurality of locations in the external memory from a second portion of the neural network output, wherein the second portion includes a content-based subportion, and wherein determining each of the one or more sets of writing weights comprises:

for each of the plurality of locations in the external memory:

computing a similarity measure between (i) a key vector derived from the content-based subportion and (ii) a respective value vector stored in the location, and

determining a respective content-based writing weight based on the computed similarity measure, and

adjusting the content-based writing weights using preceding writing weights assigned to the plurality of locations to generate the one or more sets of writing weights;

writing data defined by a third portion of the neural network output to the external memory in accordance with the sets of writing weights;

determining one or more sets of reading weights for each of the plurality of locations in the external memory from a fourth portion of the neural network output;

reading data from the external memory in accordance with the sets of reading weights; and

combining the data read from the external memory with a next system input in the sequence of system inputs to generate a next neural network input in the sequence of neural network inputs.

2. The method of claim 1 , further comprising, for each of the neural network outputs:

determining one or more sets of erasing weights for each of the plurality of locations in the external memory from a fifth portion of the neural network output; and

erasing data from the external memory in accordance with the sets of erasing weights.

3. The method of claim 2 , wherein the sets of erasing weights are the same as the sets of writing weights and the second portion is the same as the fifth portion.

4. The method of claim 1 , wherein determining each of the one or more sets of writing weights further comprises:

determining a set of location-based writing weights; and

adjusting the content-based writing weights using the location-based writing weights to generate the one or more sets of writing weights.

5. The method of claim 1 , wherein determining each of the one or more sets of reading weights comprises:

determining a set of content-based reading weights from the fourth portion of the neural network output.

6. The method of claim 5 , wherein determining each of the one or more sets of reading weights further comprises:

determining a set of location-based reading weights; and

adjusting the content-based reading weights using the location-based reading weights to generate the set of reading weights.

7. The method of claim 1 , wherein reading data from the external memory in accordance with the sets of reading weights comprises, for each set of reading weights:

determining a weighted average of values stored in the plurality of locations in the external memory in accordance with the reading weights in the set of reading weights.

8. The method of claim 1 , wherein writing data defined by the third portion of the neural network output to the external memory in accordance with the sets of writing weights comprises, for each of the sets of writing weights:

determining, from the third portion of the neural network output, a write vector for the set of writing weights; and

writing the write vector to the plurality of locations in accordance with the set of writing weights.

9. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for processing a sequence of system inputs to generate a sequence of system outputs using an augmented neural network system comprising a neural network and an external memory, wherein the neural network is configured to receive a sequence of neural network inputs and to process each neural network input to generate a neural network output from the neural network input, wherein the external memory is external to the neural network and is configured to store a respective value vector in each of a plurality of locations in the external memory, and wherein the operations comprise, for each neural network output:

providing an output derived from a first portion of the neural network output as a system output in the sequence of system outputs;

determining one or more sets of writing weights for each of the plurality of locations in the external memory from a second portion of the neural network output, wherein the second portion includes a content-based subportion, and wherein determining each of the one or more sets of writing weights comprises:

for each of the plurality of locations in the external memory:

computing a similarity measure between (i) a key vector derived from the content-based subportion and (ii) a respective value vector stored in the location, and

determining a respective content-based writing weight based on the computed similarity measure, and

adjusting the content-based writing weights using preceding writing weights assigned to the plurality of locations to generate the one or more sets of writing weights;

writing data defined by a third portion of the neural network output to the external memory in accordance with the sets of writing weights;

determining one or more sets of reading weights for each of the plurality of locations in the external memory from a fourth portion of the neural network output;

reading data from the external memory in accordance with the sets of reading weights; and

combining the data read from the external memory with a next system input in the sequence of system inputs to generate a next neural network input in the sequence of neural network inputs.

10. The one or more non-transitory computer storage media of claim 9 , wherein the operations further comprises, for each of the neural network outputs:

determining one or more sets of erasing weights for each of the plurality of locations in the external memory from a fifth portion of the neural network output; and

erasing data from the external memory in accordance with the sets of erasing weights.

11. The one or more non-transitory computer storage media of claim 9 , wherein the operations for determining each of the one or more sets of writing weights further comprises:

determining a set of location-based writing weights; and

adjusting the content-based writing weights using the location-based writing weights to generate the one or more sets of writing weights.

12. The one or more non-transitory computer storage media of claim 9 , wherein the operations for determining each of the one or more sets of reading weights comprises:

determining a set of content-based reading weights from the fourth portion of the neural network output.

13. The one or more non-transitory computer storage media of claim 12 , wherein determining each of the one or more sets of reading weights further comprises:

determining a set of location-based reading weights; and

adjusting the content-based reading weights using the location-based reading weights to generate the set of reading weights.

14. A system comprising one or more computers and one or more non-transitory computer storage media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for processing a sequence of system inputs to generate a sequence of system outputs using an augmented neural network system comprising a neural network and an external memory, wherein the neural network is configured to receive a sequence of neural network inputs and to process each neural network input to generate a neural network output from the neural network input, wherein the external memory is external to the neural network and is configured to store a respective value vector in each of a plurality of locations in the external memory, and wherein the operations comprise, for each neural network output:

providing an output derived from a first portion of the neural network output as a system output in the sequence of system outputs;

determining one or more sets of writing weights for each of the plurality of locations in the external memory from a second portion of the neural network output, wherein the second portion includes a content-based subportion, and wherein determining each of the one or more sets of writing weights comprises:

for each of the plurality of locations in the external memory:

computing a similarity measure between (i) a key vector derived from the content-based subportion and (ii) a respective value vector stored in the location, and

determining a respective content-based writing weight based on the computed similarity measure, and

adjusting the content-based writing weights using preceding writing weights assigned to the plurality of locations to generate the one or more sets of writing weights;

writing data defined by a third portion of the neural network output to the external memory in accordance with the sets of writing weights;

determining one or more sets of reading weights for each of the plurality of locations in the external memory from a fourth portion of the neural network output;

reading data from the external memory in accordance with the sets of reading weights; and

combining the data read from the external memory with a next system input in the sequence of system inputs to generate a next neural network input in the sequence of neural network inputs.

15. The system of claim 14 , wherein the operations further comprise, for each of the neural network outputs:

determining one or more sets of erasing weights for each of the plurality of locations in the external memory from a fifth portion of the neural network output; and

erasing data from the external memory in accordance with the sets of erasing weights.

16. The system of claim 14 , wherein the operations for determining each of the one or more sets of writing weights further comprises:

determining a set of location-based writing weights; and

adjusting the content-based writing weights using the location-based writing weights to generate the one or more sets of writing weights.

17. The system of claim 14 , wherein the operations for determining each of the one or more sets of reading weights comprises:

determining a set of content-based reading weights from the fourth portion of the neural network output.

18. The system of claim 17 , wherein the operations for determining each of the one or more sets of reading weights further comprises:

determining a set of location-based reading weights; and

adjusting the content-based reading weights using the location-based reading weights to generate the set of reading weights.

19. The system of claim 14 , wherein the operations for reading data from the external memory in accordance with the sets of reading weights comprises, for each set of reading weights:

determining a weighted average of values stored in the plurality of locations in the external memory in accordance with the reading weights in the set of reading weights.

20. The system of claim 14 , wherein the operations for writing data defined by the third portion of the neural network output to the external memory in accordance with the sets of writing weights comprises, for each of the sets of writing weights:

determining, from the third portion of the neural network output, a write vector for the set of writing weights; and

writing the write vector to the plurality of locations in accordance with the set of writing weights.