Generating output data items using template data items

1. A neural network system implemented by one or more computers, the neural network system comprising:

one or more memories storing a plurality of template data items, each of the plurality of template data items corresponding to a respective memory address;

one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

receiving an input data item;

selecting a memory address based upon the received input data item; and

retrieving the template data item corresponding to the selected memory address from the memory;

providing the retrieved template data item retrieved from the memory and the received input data item as input to an encoder neural network that is configured to process the received input data item and the retrieved template data item to generate a latent variable representation that represents latent factors of variation that account for variability in the input data item relative to the retrieved template data item;

obtaining as output from the encoder neural network the latent variable representation that represents latent factors of variation that account for variability in the input data item relative to the retrieved template data item;

providing the latent variable representation and the retrieved template data item as input to a decoder neural network configured to process the retrieved template data item and the latent variable representation to generate an output data item; and

obtaining as output from the decoder neural network the output data item.

2. The system of claim 1 , wherein selecting a memory address comprises selecting the memory address based upon a respective similarity score for each of the plurality of template data items stored in the memory that measure similarity between the template data item and the received input data item.

3. The system of claim 2 , the operations further comprising:

providing the received input data item as input to a first embedding neural network configured to process the received input data input to generate a first representation of the input data item;

providing each of the template data items as input to a second embedding neural network configured to process each of the template data items to generate a respective second representation of each of the template data items; and

wherein selecting the memory address comprising computing the respective similarity scores based upon the first representation of the input data item and the respective second representations of the plurality of template data items stored in the memory.

4. The system of claim 2 , wherein selecting the memory address further comprises selecting the memory address by sampling from a probability distribution over the respective similarity scores.

5. The system of claim 2 , wherein the respective similarity score is based upon one of the following: an inner product, a cosine similarity and an L2-distance.

6. The system of claim 1 , wherein the plurality of template data items are generated based upon a training data set.

7. The system of claim 6 , wherein the plurality of template data items are learned from the training data set.

8. The system of claim 6 , wherein the plurality of template data items are generated based upon a selection of data items from the training data set, the selection based upon a classification of the data.

9. The system of claim 3 , wherein the first embedding neural network and the second embedding neural network share at least a portion of their respective weights.

10. The system of claim 1 , wherein the encoder neural network and the decoder neural network are based upon a variational autoencoder framework.

11. The system of claim 10 , wherein the encoder neural network comprises an encoder portion of a variational autoencoder and the decoder neural network comprises a decoder portion of the variational autoencoder.

12. The system of claim 10 , wherein the encoder neural network and the decoder neural network are trained based upon optimizing a variational lower bound.

13. The system of claim 1 , wherein the output data item is of the same classification as the input data item.

14. The system of claim 1 , wherein the input data item is a data item of a category selected from the group consisting of: image data, video data, motion data, speech data, audio data, an electronic document, data representing a state of an environment and data representing an action.

15. The system of claim 1 , wherein one or more of the encoder neural network or the decoder neural network comprises a recurrent neural network.

16. A method of generating a data item, comprising:

receiving, by one or more computers, an input data item;

selecting, by the one or more computers, a memory address based upon the received input data item, the selected memory address corresponding to one of a plurality of template data items stored in one or more memories;

retrieving, by the one or more computers and from the one or more memories, the template data item corresponding to the selected memory address;

providing the retrieved template data item retrieved from the memory and the received input data item as input to an encoder neural network that is configured to process the received input data item and the retrieved template data item to generate a latent variable representation that represents latent factors of variation that account for variability in the input data item relative to the retrieved template data item;

obtaining as output from the encoder neural network the latent variable representation that represents latent factors of variation that account for variability in the input data item relative to the retrieved template data item;

providing the latent variable representation and the retrieved template data item as input to a decoder neural network configured to process the retrieved template data item and the latent variable representation to generate an output data item; and

obtaining as output from the decoder neural network the output data item.

17. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement a neural network system, the neural network system comprising:

one or more memories storing a plurality of template data items, each of the plurality of template data items corresponding to a respective memory address;

one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

receiving an input data item;

selecting a memory address based upon the received input data item; and

retrieving the template data item corresponding to the selected memory address from the memory;

providing the retrieved template data item retrieved from the memory and the received input data item as input to an encoder neural network that is configured to process the received input data item and the retrieved template data item to generate a latent variable representation that represents latent factors of variation that account for variability in the input data item relative to the retrieved template data item;

obtaining as output from the encoder neural network the latent variable representation that represents latent factors of variation that account for variability in the input data item relative to the retrieved template data item;

providing the latent variable representation and the retrieved template data item as input to a decoder neural network configured to process the retrieved template data item and the latent variable representation to generate an output data item; and

obtaining as output from the decoder neural network the output data item.

18. The method of claim 16 , wherein selecting a memory address comprises selecting the memory address based upon a respective similarity score for each of the plurality of template data items stored in the memory that measure similarity between the template data item and the received input data item.

19. The method of claim 18 , further comprising:

providing the received input data item as input to a first embedding neural network configured to process the received input data input to generate a first representation of the input data item;

providing each of the template data items as input to a second embedding neural network configured to process each of the template data items to generate a respective second representation of each of the template data items; and

wherein selecting the memory address comprising computing the respective similarity scores based upon the first representation of the input data item and the respective second representations of the plurality of template data items stored in the memory.

20. The method of claim 18 , wherein selecting the memory address further comprises selecting the memory address by sampling from a probability distribution over the respective similarity scores.