Learning visual concepts using neural networks

1. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement:

an image decoder neural network having a plurality of image decoder parameters, wherein the image decoder neural network is configured to:

receive an image decoder input comprising a respective value for each of a plurality of generative visual factors of variation; and

process the image decoder input in accordance with the image decoder parameters to generate an output image defined by the values for the generative visual factors of variation in the image decoder input;

a symbol encoder neural network having a plurality of symbol encoder parameters, wherein the symbol encoder neural network is configured to:

receive a symbol input comprising one or more symbols from a vocabulary of symbols; and

process the symbol input in accordance with the symbol encoder parameters to generate a symbol encoder output that parameterizes a distribution over possible values for each of the plurality of generative visual factors of variation;

a symbol recombination neural network having a plurality of recombination parameters, the symbol recombination neural network configured to:

receive a recombination input comprising (i) a first symbol encoder output generated from a first symbol input, (ii) a second symbol encoder output generated from a second symbol input, and (iii) data identifying a recombination operator to be applied to the first symbol input and the second symbol input; and

process the recombination input in accordance with the recombination parameters to generate a recombined output that parametrizes a recombined distribution over possible values for each of the plurality of generative visual factors of variation;

a subsystem configured to:

process the first symbol input using the symbol encoder neural network to generate the first symbol encoder output;

process the second symbol input using the symbol encoder neural network to generate the second symbol encoder output;

process the recombined input using the symbol recombination neural network to generate the recombined output; and

generate a first output image that depicts concepts referred to by a recombined symbol resulting from applying the recombination operator to the first symbol input and the second symbol input, by:

sampling, from the recombined distribution parameterized by the recombined output, a respective value for each of the plurality of generative visual factors of variation; and

processing a first image decoder input comprising the respective values for the generative visual factors of variation using the image decoder neural network to generate the first output image.

2. The system of claim 1 , wherein the instructions further cause the one or more computers to implement:

an image encoder neural network having a plurality of image encoder parameters, wherein the image encoder neural network is configured to:

receive an input image; and

process the input image in accordance with the image encoder parameters to generate an image encoder output that parameterizes a distribution over possible values for each of a plurality of generative visual factors of variation; and

a symbol decoder neural network having a plurality of symbol decoder parameters, wherein the symbol decoder neural network is configured to:

receive a symbol decoder input comprising a respective value for each of the plurality of generative visual factors; and

process the symbol decoder input in accordance with the symbol decoder parameters to generate a symbol output that includes one or more symbols from the vocabulary of symbols, and wherein the subsystem is further configured to:

receive a new input image; and

generate a new symbol output that includes one or more symbols that refer to concepts depicted in the new input image, comprising:

processing the new input image using the image encoder neural network to generate a new image encoder output for the new input image;

sampling, from the distribution parameterized by the new image encoder output, a respective value for each of the plurality of generative visual factors of variation; and

processing a new symbol decoder input comprising the respective values for the generative visual factors of variation using the symbol decoder neural network to generate the new symbol output.

3. The system of claim 1 , wherein the recombination neural network includes at least one layer that applies a recombination dependent transformation to an activation of the layer.

4. The system of claim 3 , wherein the recombination dependent transformation is a recombination dependent bias.

5. The system of claim 1 , wherein the recombination neural network is a conditional convolutional module that is conditioned on the operator.

6. The system of claim 5 , wherein the convolutional module is conditioned on the operator using a tensor product operation.

7. The system of claim 2 , wherein the symbol encoder neural network and the symbol decoder neural networks comprise feedforward neural networks.

8. The system of claim 2 , wherein the symbol encoder neural network and the symbol decoder neural networks comprise recurrent neural networks.

9. The system of claim 2 , wherein the image encoder neural network has been trained to generate disentangled representations of the plurality of generative factors of variation.

10. The system of claim 1 , wherein the subsystem is further configured to:

receive a new symbol input comprising one or more symbols from the vocabulary; and

generate a new output image that depicts concepts referred to by the new symbol input, comprising:

processing the new symbol input using the symbol encoder neural network to generate a new symbol encoder output for the new symbol input;

sampling, from the distribution parameterized by the new symbol encoder output, a respective value for each of the plurality of generative visual factors of variation; and

processing a new image decoder input comprising the respective values for the generative visual factors of variation using the image decoder neural network to generate the new output image.

11. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement:

an image decoder neural network having a plurality of image decoder parameters, wherein the image decoder neural network is configured to:

receive an image decoder input comprising a respective value for each of a plurality of generative visual factors of variation; and

process the image decoder input in accordance with the image decoder parameters to generate an output image defined by the values for the generative visual factors of variation in the image decoder input;

a symbol encoder neural network having a plurality of symbol encoder parameters, wherein the symbol encoder neural network is configured to:

receive a symbol input comprising one or more symbols from a vocabulary of symbols; and

process the symbol input in accordance with the symbol encoder parameters to generate a symbol encoder output that parameterizes a distribution over possible values for each of the plurality of generative visual factors of variation;

a symbol recombination neural network having a plurality of recombination parameters, the symbol recombination neural network configured to:

receive a recombination input comprising (i) a first symbol encoder output generated from a first symbol input, (ii) a second symbol encoder output generated from a second symbol input, and (iii) data identifying a recombination operator to be applied to the first symbol input and the second symbol input; and

process the recombination input in accordance with the recombination parameters to generate a recombined output that parametrizes a recombined distribution over possible values for each of the plurality of generative visual factors of variation;

and

a subsystem configured to:

process the first symbol input using the symbol encoder neural network to generate the first symbol encoder output;

process the second symbol input using the symbol encoder neural network to generate the second symbol encoder output;

process the recombination input using the symbol recombination neural network to generate the recombined output; and

generate a first output image that depicts concepts referred to by a recombined symbol resulting from applying the recombination operator to the first symbol input and the second symbol input, by:

sampling, from the recombined distribution parameterized by the recombined output, a respective value for each of the plurality of generative visual factors of variation; and

processing a first image decoder input comprising the respective values for the generative visual factors of variation using the image decoder neural network to generate the first output image.

12. A method performed by one or more computers, the method comprising:

receiving a first symbol input comprising one or more symbols from a vocabulary of symbols;

receiving a second symbol input comprising one or more symbols from the vocabulary of symbols;

receiving data specifying a recombination operator to be applied to the first symbol input and the second symbol input;

processing the first symbol input using a symbol encoder neural network having symbol encoder parameters, wherein the symbol encoder neural network is configured to process the first symbol input in accordance with the symbol encoder parameters to generate a first symbol encoder output that parameterizes a first distribution over possible values for each of a plurality of generative visual factors of variation;

processing the second symbol input using the symbol encoder neural network, wherein the symbol encoder neural network is configured process the second symbol input in accordance with the symbol encoder parameters to generate a second symbol encoder output that parameterizes a second distribution over possible values for each of the plurality of generative visual factors of variation;

processing a recombination input comprising (i) the first symbol encoder output, (ii) the second symbol encoder output, and (iii) data identifying the recombination operator to be applied to the first symbol input and the second symbol input using a symbol recombination neural network having recombination parameters, wherein the symbol recombination neural network is configured to process the recombination input in accordance with the recombination parameters to generate a recombined output that parametrizes a recombined distribution over possible values for each of the plurality of generative visual factors of variation; and

generating a first output image that depicts concepts referred to by a recombined symbol resulting from applying the recombination operator to the first symbol input and the second symbol input, by:

sampling, from the recombined distribution parameterized by the recombined output, a respective value for each of the plurality of generative visual factors of variation; and

processing a first image decoder input comprising the respective values for the generative visual factors of variation using an image decoder neural network having a plurality of image decoder parameters, wherein the image decoder neural network is configured to receive the first image decoder input and process the first image decoder input in accordance with the image decoder parameters to generate the first output image.

13. The method of claim 12 , further comprising:

receiving a new input image; and

generating a new symbol output that includes one or more symbols that refer to concepts depicted in the new input image, comprising:

processing the new input image using an image encoder neural network having a plurality of image encoder parameters, wherein the image encoder neural network is configured to process the new input image in accordance with the image encoder parameters to generate a new image encoder output that parameterizes a distribution over possible values for each of the plurality of generative visual factors of variation;

sampling, from the distribution parameterized by the new image encoder output, a respective value for each of the plurality of generative visual factors of variation; and

processing a new symbol decoder input comprising the respective values for the generative visual factors of variation using a symbol decoder neural network having a plurality of symbol decoder parameters, wherein the symbol decoder neural network is configured to process the new symbol decoder input in accordance with the symbol decoder parameters to generate the new symbol output.

14. The method of claim 12 , wherein the recombination neural network includes at least one layer that applies a recombination dependent transformation to an activation of the layer.

15. The method of claim 14 , wherein the recombination dependent transformation is a recombination dependent bias.

16. The method of claim 12 , wherein the recombination neural network is a conditional convolutional module that is conditioned on the operator.

17. The method of claim 16 , wherein the convolutional module is conditioned on the operator using a tensor product operation.

18. The method of claim 13 , wherein the symbol encoder neural network and the symbol decoder neural networks comprise feedforward neural networks.

19. The method of claim 13 , wherein the symbol encoder neural network and the symbol decoder neural networks comprise recurrent neural networks.

20. The method of claim 12 , further comprising:

receiving a new symbol input comprising one or more symbols from the vocabulary; and

generating a new output image that depicts concepts referred to by the new symbol input, comprising:

processing the new symbol input using the symbol encoder neural network to generate a new symbol encoder output for the new symbol input;

sampling, from the distribution parameterized by the new symbol encoder output, a respective value for each of the plurality of generative visual factors of variation; and

processing a new image decoder input comprising the respective values for the generative visual factors of variation using the image decoder neural network to generate the new output image.