Learning visual concepts using neural networks

1. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a symbol encoder neural network and a symbol decoder neural network using a trained

image encoder neural network having a plurality of image encoder parameters, wherein the image encoder neural network is configured to:

receive an input image; and

process the input image in accordance with the image encoder parameters to generate an image encoder output that parameterizes a distribution over possible values for each of a plurality of generative visual factors of variation;

wherein the symbol encoder neural network has a plurality of symbol encoder parameters, and is configured to:

receive a symbol input comprising one or more symbols from a vocabulary of symbols; and

process the symbol input in accordance with the symbol encoder parameters to generate a symbol encoder output that parameterizes a distribution over possible values for each of the plurality of generative visual factors of variation;

wherein the symbol decoder neural network has a plurality of symbol decoder parameters, and is configured to:

receive a symbol decoder input comprising a respective value for each of the plurality of generative visual factors; and

process the symbol decoder input in accordance with the symbol decoder parameters to generate a symbol output that includes one or more symbols from the vocabulary of symbols; and

wherein the training comprises determining trained values of the symbol encoder parameters and the symbol decoder parameters by repeatedly:

receiving a training symbol input comprising one or more symbols from the vocabulary and a training image that matches the training symbol input;

processing the training image using the image encoder neural network in accordance with the image encoder parameters to generate a training image encoder output for the training image;

processing the training symbol input using the symbol encoder neural network in accordance with current values of the symbol encoder parameters to generate a training symbol encoder output for the training symbol input;

sampling, from a distribution parameterized by the training symbol encoder output, a respective value for each of the plurality of visual factors;

processing a training symbol decoder input comprising the respective values for the visual factors using the symbol decoder neural network in accordance with current values of the symbol decoder parameters to generate a training symbol output;

determining a gradient with respect to the symbol encoder parameters and the symbol decoder parameters of an objective function that includes (i) a variational auto encoder (VAE) objective and (ii) a term that encourages alignment between the training symbol encoder output and the training image encoder output; and

determining an update to the current values of the symbol encoder parameters and the symbol decoder parameters using the gradient.

2. The system of claim 1 , the operations further comprising, prior to determining the trained values of the symbol encoder parameters and the symbol decoder parameters:

training the image encoder neural network to generate disentangled representations of the factors to determine the first values of the image encoder parameters.

3. The system of claim 2 , wherein training the image encoder neural network to generate disentangled representations of the factors to determine the first values of the image encoder parameters comprises:

training the image encoder neural network and an image decoder neural network jointly using a β-VAE training technique;

the image decoder neural network having a plurality of image decoder parameters, wherein the image decoder neural network is configured to:

receive an image decoder input comprising a respective value for each of the plurality of visual factors; and

process the image decoder input in accordance with the image decoder parameters to generate an output image defined by the values for the visual factors in the image decoder input.

4. The system of claim 3 , wherein training the image encoder neural network to generate disentangled representations of the factors to determine the first values of the image encoder parameters comprises:

training the image encoder neural network and the image decoder neural network jointly using a β-VAE training technique that replaces a pixel level log-likelihood with a loss in a high-level feature space of a denoising autoencoder.

5. The system of claim 2 , wherein the term that encourages alignment is a KL divergence between (i) the training image encoder output and (ii) the training symbol encoder output.

6. The system of claim 1 , wherein the operations further comprise training a symbol recombination neural network having a plurality of recombination parameters, the symbol recombination neural network configured to:

receive a recombination input comprising (i) a first symbol encoder output generated from a first symbol input, (ii) a second symbol encoder output generated from a second symbol input, and (iii) data identifying a recombination operator to be applied to the first symbol input and the second symbol input; and

process the recombination input in accordance with the recombination parameters to generate a recombined output that parametrizes a recombined distribution over possible values for each of the plurality of generative visual factors of variation.

7. The system of claim 6 , wherein training the symbol recombination neural network comprises:

obtaining a training first symbol input, a training second symbol input, data identifying a recombination operator, and a training image that depicts concepts referred to by a recombination of the training first symbol input and the training second symbol input using the recombination operator;

processing the training first symbol input using the symbol encoder neural network to generate a training first symbol encoder output;

processing the training second symbol input using the symbol encoder neural network to generate a training second symbol encoder output;

processing a training recombined input comprising the training first symbol encoder output, the training second symbol encoder output, and data identifying the recombination operator using the symbol recombination neural network in accordance with current values of the recombination parameters to generate a training recombined output;

processing the training image using the image encoder neural network to generate a training image encoder output;

determining a gradient of an objective function that depends on a divergence between the training image encoder output and the training recombined output; and

determining an update to the current values of the recombination parameters using the gradient.

8. One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a symbol encoder neural network and a symbol decoder neural network using a trained image encoder neural network having a plurality of image encoder parameters, wherein the image encoder neural network is configured to:

receive an input image; and

process the input image in accordance with the image encoder parameters to generate an image encoder output that parameterizes a distribution over possible values for each of a plurality of generative visual factors of variation;

wherein the symbol encoder neural network has a plurality of symbol encoder parameters and is configured to:

receive a symbol input comprising one or more symbols from a vocabulary of symbols; and

process the symbol input in accordance with the symbol encoder parameters to generate a symbol encoder output that parameterizes a distribution over possible values for each of the plurality of generative visual factors of variation;

wherein the symbol decoder neural network has a plurality of symbol decoder parameters and is configured to:

receive a symbol decoder input comprising a respective value for each of the plurality of generative visual factors; and

process the symbol decoder input in accordance with the symbol decoder parameters to generate a symbol output that includes one or more symbols from the vocabulary of symbols; and

wherein the training comprises determining trained values of the symbol encoder parameters and the symbol decoder parameters by repeatedly:

receiving a training symbol input comprising one or more symbols from the vocabulary and a training image that matches the training symbol input;

processing the training image using the image encoder neural network in accordance with the image encoder parameters to generate a training image encoder output for the training image;

processing the training symbol input using the symbol encoder neural network in accordance with current values of the symbol encoder parameters to generate a training symbol encoder output for the training symbol input;

sampling, from a distribution parameterized by the training symbol encoder output, a respective value for each of the plurality of visual factors;

processing a training symbol decoder input comprising the respective values for the visual factors using the symbol decoder neural network in accordance with current values of the symbol decoder parameters to generate a training symbol output;

determining a gradient with respect to the symbol encoder parameters and the symbol decoder parameters of an objective function that includes (i) a variational auto encoder (VAE) objective and (ii) a term that encourages alignment between the training symbol encoder output and the training image encoder output; and

determining an update to the current values of the symbol encoder parameters and the symbol decoder parameters using the gradient.

9. The one or more non-transitory computer-readable storage media of claim 8 , the operations further comprising, prior to determining the trained values of the symbol encoder parameters and the symbol decoder parameters:

training the image encoder neural network to generate disentangled representations of the factors to determine the first values of the image encoder parameters.

10. The one or more non-transitory computer-readable storage media of claim 9 , wherein training the image encoder neural network to generate disentangled representations of the factors to determine the first values of the image encoder parameters comprises:

training the image encoder neural network and an image decoder neural network jointly using a β-VAE training technique;

the image decoder neural network having a plurality of image decoder parameters, wherein the image decoder neural network is configured to:

receive an image decoder input comprising a respective value for each of the plurality of visual factors; and

process the image decoder input in accordance with the image decoder parameters to generate an output image defined by the values for the visual factors in the image decoder input.

11. The one or more non-transitory computer-readable storage media of claim 10 , wherein training the image encoder neural network to generate disentangled representations of the factors to determine the first values of the image encoder parameters comprises:

training the image encoder neural network and the image decoder neural network jointly using a β-VAE training technique that replaces a pixel level log-likelihood with a loss in a high-level feature space of a denoising autoencoder.

12. The one or more non-transitory computer-readable storage media of claim 9 , wherein the term that encourages alignment is a KL divergence between (i) the training image encoder output and (ii) the training symbol encoder output.

13. The one or more non-transitory computer-readable storage media of claim 8 , wherein the operations further comprise training a symbol recombination neural network having a plurality of recombination parameters, the symbol recombination neural network configured to:

receive a recombination input comprising (i) a first symbol encoder output generated from a first symbol input, (ii) a second symbol encoder output generated from a second symbol input, and (iii) data identifying a recombination operator to be applied to the first symbol input and the second symbol input; and

process the recombination input in accordance with the recombination parameters to generate a recombined output that parametrizes a recombined distribution over possible values for each of the plurality of generative visual factors of variation.

14. A method performed by one or more computers and for training a symbol encoder neural network and a symbol decoder neural network using a trained image encoder neural network having a plurality of image encoder parameters, wherein the image encoder neural network is configured to:

receive an input image; and

process the input image in accordance with the image encoder parameters to generate an image encoder output that parameterizes a distribution over possible values for each of a plurality of generative visual factors of variation;

wherein the symbol encoder neural network has a plurality of symbol encoder parameters and is configured to:

receive a symbol input comprising one or more symbols from a vocabulary of symbols; and

process the symbol input in accordance with the symbol encoder parameters to generate a symbol encoder output that parameterizes a distribution over possible values for each of the plurality of generative visual factors of variation;

wherein the symbol decoder neural network has a plurality of symbol decoder parameters and is configured to:

receive a symbol decoder input comprising a respective value for each of the plurality of generative visual factors; and

process the symbol decoder input in accordance with the symbol decoder parameters to generate a symbol output that includes one or more symbols from the vocabulary of symbols; and

wherein the method comprises determining trained values of the symbol encoder parameters and the symbol decoder parameters by repeatedly:

receiving a training symbol input comprising one or more symbols from the vocabulary and a training image that matches the training symbol input;

processing the training image using the image encoder neural network in accordance with the image encoder parameters to generate a training image encoder output for the training image;

processing the training symbol input using the symbol encoder neural network in accordance with current values of the symbol encoder parameters to generate a training symbol encoder output for the training symbol input;

sampling, from a distribution parameterized by the training symbol encoder output, a respective value for each of the plurality of visual factors;

processing a training symbol decoder input comprising the respective values for the visual factors using the symbol decoder neural network in accordance with current values of the symbol decoder parameters to generate a training symbol output;

determining a gradient with respect to the symbol encoder parameters and the symbol decoder parameters of an objective function that includes (i) a variational auto encoder (VAE) objective and (ii) a term that encourages alignment between the training symbol encoder output and the training image encoder output; and

determining an update to the current values of the symbol encoder parameters and the symbol decoder parameters using the gradient.

15. The method of claim 14 , the operations further comprising, prior to determining the trained values of the symbol encoder parameters and the symbol decoder parameters:

training the image encoder neural network to generate disentangled representations of the factors to determine the first values of the image encoder parameters.

16. The method of claim 15 , wherein training the image encoder neural network to generate disentangled representations of the factors to determine the first values of the image encoder parameters comprises:

training the image encoder neural network and an image decoder neural network jointly using a β-VAE training technique;

the image decoder neural network having a plurality of image decoder parameters, wherein the image decoder neural network is configured to:

receive an image decoder input comprising a respective value for each of the plurality of visual factors; and

process the image decoder input in accordance with the image decoder parameters to generate an output image defined by the values for the visual factors in the image decoder input.

17. The method of claim 16 , wherein training the image encoder neural network to generate disentangled representations of the factors to determine the first values of the image encoder parameters comprises:

training the image encoder neural network and the image decoder neural network jointly using a β-VAE training technique that replaces a pixel level log-likelihood with a loss in a high-level feature space of a denoising autoencoder.

18. The method of claim 15 , wherein the term that encourages alignment is a KL divergence between (i) the training image encoder output and (ii) the training symbol encoder output.

19. The method of claim 14 , wherein the operations further comprise training a symbol recombination neural network having a plurality of recombination parameters, the symbol recombination neural network configured to:

receive a recombination input comprising (i) a first symbol encoder output generated from a first symbol input, (ii) a second symbol encoder output generated from a second symbol input, and (iii) data identifying a recombination operator to be applied to the first symbol input and the second symbol input; and

process the recombination input in accordance with the recombination parameters to generate a recombined output that parametrizes a recombined distribution over possible values for each of the plurality of generative visual factors of variation.

20. The method of claim 19 , wherein training the symbol recombination neural network comprises:

obtaining a training first symbol input, a training second symbol input, data identifying a recombination operator, and a training image that depicts concepts referred to by a recombination of the training first symbol input and the training second symbol input using the recombination operator;

processing the training first symbol input using the symbol encoder neural network to generate a training first symbol encoder output;

processing the training second symbol input using the symbol encoder neural network to generate a training second symbol encoder output;

processing a training recombined input comprising the training first symbol encoder output, the training second symbol encoder output, and data identifying the recombination operator using the symbol recombination neural network in accordance with current values of the recombination parameters to generate a training recombined output;

processing the training image using the image encoder neural network to generate a training image encoder output;

determining a gradient of an objective function that depends on a divergence between the training image encoder output and the training recombined output; and

determining an update to the current values of the recombination parameters using the gradient.