Generative neural networks
View Patent ↗Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for a neural network system. In one aspect, a neural network system includes a recurrent neural network that is configured to, for each time step of a predetermined number of time steps, receive a set of latent variables for the time step and process the latent variables to update a hidden state of the recurrent neural network; and a generative subsystem that is configured to, for each time step, generate the set of latent variables for the time step and provide the set of latent variables as input to the recurrent neural network; update a hidden canvas using the updated hidden state of the recurrent neural network; and, for a last time step, generate an output image using the updated hidden canvas for the last time step.
1. A neural network system implemented by one or more computers, the neural network system comprising:
a recurrent neural network that is configured to, for each time step of a predetermined number of time steps, receive a set of latent variables for the time step and process the set of latent variables to update a hidden state of the recurrent neural network; and
a generative subsystem that is configured to:
for each time step of the predetermined number of time steps:
generate the set of latent variables for the time step and provide the set of latent variables as input to the recurrent neural network;
update a hidden canvas using the updated hidden state of the recurrent neural network; and
for a last time step of the predetermined number of time steps:
generate an output image using the updated hidden canvas for the last time step.
2. The neural network system of claim 1 , wherein the recurrent neural network is a long short-term memory (LSTM) neural network.
3. The neural network system of claim 1 , wherein updating the hidden canvas using the updated hidden state of the recurrent neural network comprises:
generating a write for the time step by processing the hidden state using a writing mechanism; and
applying a canvas function to the write to combine the write with the hidden canvas to generate the updated hidden canvas for the time step.
4. The neural network system of claim 3 , wherein the writing mechanism is an attention-based writing mechanism.
5. The neural network system of claim 4 , wherein the attention-based writing mechanism uses a spatial transformers function.
6. The neural network system of claim 3 , wherein the canvas function is an additive canvas function.
7. The neural network system of claim 3 , wherein the canvas function is a gated recurrent canvas function.
8. The neural network system of claim 1 , wherein generating an output image using the updated hidden canvas for the last time step comprises:
mapping the updated hidden canvas for the last time step to sufficient statistics of a likelihood model using an observation function; and
generating the output image by sampling from the likelihood model.
9. The neural network system of claim 1 , wherein generating the set of latent variables for the time step comprises sampling the latent variables from a prior distribution.
10. The neural network system of claim 1 , wherein the output image is a reconstruction of an input image, and wherein the neural network system further comprises:
an inference subsystem, wherein the inference subsystem is configured to, for each of the predetermined number of time steps:
generate a sprite representation of the input image by applying a reading function to the input image and the hidden state of the recurrent neural network;
determine parameters of a distribution of latent variables using the sprite representation and the hidden state of the recurrent neural network; and
sample the set of latent variables for the time step from the distribution in accordance with the determined parameters.
11. A method comprising:
at a generative subsystem of a neural network system implemented by one or more computers,
for each time step of a predetermined number of time steps:
receiving, from a recurrent neural network of the neural network system, an updated hidden state of the recurrent neural network;
generating a set of latent variables for the time step and providing the set of latent variables as input to the recurrent neural network; and
updating a hidden canvas using the updated hidden state of the recurrent neural network; and
for a last time step of the predetermined number of time steps, generating an output image using the updated hidden canvas for the last time step.
12. The method of claim 11 , wherein the recurrent neural network is a long short-term memory (LSTM) neural network.
13. The method of claim 11 , wherein updating the hidden canvas using the updated hidden state of the recurrent neural network comprises:
generating a write for the time step by processing the hidden state using a writing mechanism; and
applying a canvas function to the write to combine the write with the hidden canvas to generate the updated hidden canvas for the time step.
14. The method of claim 13 , wherein the writing mechanism is an attention-based writing mechanism.
15. The method of claim 14 , wherein the attention-based writing mechanism uses a spatial transformers function.
16. The method of claim 13 , wherein the canvas function is an additive canvas function or is a gated recurrent canvas function.
17. The method of claim 11 , wherein generating an output image using the updated hidden canvas for the last time step comprises:
mapping the updated hidden canvas for the last time step to sufficient statistics of a likelihood model using an observation function; and
generating the output image by sampling from the likelihood model.
18. The method of claim 11 , wherein the output image is a reconstruction of an input image, and wherein the method further comprises, at an inference subsystem of the neural network system,
generating a sprite representation of the input image by applying a reading function to the input image and the hidden state of the recurrent neural network;
determining parameters of a distribution of latent variables using the sprite representation and the hidden state of the recurrent neural network; and
sampling the set of latent variables for the time step from the distribution in accordance with the determined parameters.
19. A non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
at a generative subsystem of a neural network system implemented by one or more computers,
for each time step of a predetermined number of time steps:
receiving, from a recurrent neural network of the neural network system, an updated hidden state of the recurrent neural network;
generating a set of latent variables for the time step and providing the set of latent variables as input to the recurrent neural network; and
updating a hidden canvas using the updated hidden state of the recurrent neural network; and
for a last time step of the predetermined number of time steps, generating an output image using the updated hidden canvas for the last time step.
20. The non-transitory computer storage medium of claim 19 , wherein the output image is a reconstruction of an input image, and wherein the operations further comprise, at an inference subsystem of the neural network system,
generating a sprite representation of the input image by applying a reading function to the input image and the hidden state of the recurrent neural network;
determining parameters of a distribution of latent variables using the sprite representation and the hidden state of the recurrent neural network; and
sampling the set of latent variables for the time step from the distribution in accordance with the determined parameters.