Controllable conditional image generation

1. A method for synthesizing an image, the method comprising

generating, using a first generator stage of an image synthesis computer system, an intermediate representation of an image, wherein the intermediate representation captures at least a particular high-level structure of the image, and wherein the first generator stage is initialized by a random noise sample to generate the intermediate representation of the image without specific condition provided for the image, wherein the random noise sample at least in part determines the particular high-level structure of the image; and

generating, using a second generator stage of the image synthesis computer system, a conditional image, wherein the conditional image is generated based on i) the intermediate representation of the image generated by the first generator stage and ii) a condition provided for the image, the condition specifying a desired style for the image, such that the conditional image is generated to match i) the particular high-level structure captured in the intermediate representation of the image and ii) the desired style specified by the condition provided for the image.

2. The method of claim 1 , wherein:

the first generator stage comprises an unconditional generative neural network, and

the second generator stage comprises a conditional generative neural network.

3. The method of claim 2 , wherein the unconditional generative neural network comprises i) a structure-prior generative sub-network trained to generate the intermediate representation of the image and ii) an unconditional generative sub-network trained to generate an unconditional image based on the intermediate representation of the image.

4. The method of claim 2 , further comprising

training the unconditional generative neural network using an unconditional generative adversarial network (GAN), and

training the conditional generative neural network using a conditional generative adversarial network (CGAN).

5. The method of claim 2 , further comprising training the unconditional generative neural network and the conditional generative neural network concurrently using semi-supervised training, wherein the unconditional generative neural network is trained using unpaired data and the conditional generative neural network is trained using paired data.

6. The method of claim 1 , wherein

the method further comprises concatenating the intermediate representation with a tensor representing a sample drawn around the condition provided for the image, and

generating the conditional image includes using the concatenated intermediate representation with the tensor as a structure-prior for generating the conditional image.

7. The method of claim 1 , further comprising

fixing the intermediate representation, and

generating a set of conditional images, wherein respective conditional images in the set of conditional images are generated based on i) the fixed intermediate representation of the image and ii) one or both of a) respective varying conditions provided for the respective conditional images and b) respective samples drawn around a fixed condition provided for the conditional images.

8. The method of claim 1 , wherein the condition provided for the image comprises one of i) a text description of the desired style for the image and ii) an attribute set having one or more attributes defining the desired style of the image.

9. An image synthesis computer system, comprising:

a first generator stage configured to generate an intermediate representation of an image, wherein the intermediate representation captures at least a particular high-level structure of the image, and wherein the first generator stage is initialized by a random noise sample to generate the intermediate representation of the image without specific condition provided for the image, wherein the random noise sample at least in part determines the particular high-level structure of the image; and

a second generator stage configured to generate a conditional image, wherein the conditional image is generated based on i) the intermediate representation of the image and ii) a condition provided for the image, the condition specifying a desired style for the image, such that the conditional image is generated to match i) the particular high-level structure captured in the intermediate representation of the image and ii) the desired style specified by the condition provided for the image.

10. The image synthesis computer system of claim 9 , wherein:

the first generator stage comprises an unconditional generative neural network, and

the second generator stage comprises a conditional generative neural network.

11. The image synthesis computer system of claim 10 , wherein the unconditional generative neural network comprises a structure-prior generative sub-network trained to generate the intermediate representation of the image and an unconditional generative sub-network trained to generate an unconditional image based on the intermediate representation of the image.

12. The image synthesis computer system of claim 10 , wherein

the unconditional generative neural network is trained using an unconditional generative adversarial network (GAN), and

the conditional generative neural network is trained using a conditional generative adversarial network (CGAN).

13. The image synthesis computer system of claim 10 , wherein the unconditional generative neural network and the conditional generative neural network are trained concurrently using semi-supervised training, wherein the unconditional generative neural network is trained using unpaired data and the conditional generative neural network is trained using paired data.

14. The image synthesis computer system of claim 10 , wherein the second generator stage is further configured to

concatenate the intermediate representation with a tensor representing a sample drawn around the condition provided for the image, and

generate the conditional image includes using the concatenated intermediate representation with the tensor as a structure-prior for generating the conditional image.

15. The image synthesis computer system of claim 9 , wherein the condition provided for the image comprises one of i) a text description of the desired style for the image and ii) an attribute set having one or more attributes defining the desired style of the image.

16. The tangible computer-readable medium or media of claim 15 , further storing machine readable instructions that, when executed by the one or more processors, cause the one or more processors to

train the unconditional generative neural network using an unconditional generative adversarial network (GAN), and

train the conditional generative neural network using a conditional generative adversarial network (CGAN).

17. The tangible computer-readable medium or media of claim 15 , further storing machine readable instructions that, when executed by the one or more processors, cause the one or more processors to train the unconditional generative neural network and the conditional generative neural network concurrently using semi-supervised training, wherein the unconditional generative neural network is trained using unpaired data and the conditional generative neural network is trained using paired data.

18. A tangible computer readable medium, or media, storing machine readable instructions that, when executed by one or more processors, cause the one or more processors to:

generate an intermediate representation of an image, wherein the intermediate representation captures at least a particular high-level structure of the image, and wherein the first generator stage is initialized by a random noise sample to generate the intermediate representation of the image without specific condition provided for the image, wherein the random noise sample at least in part determines the particular high-level structure of the image; and

generate a conditional image, wherein the conditional image is generated based on i) the intermediate representation of the image and ii) a condition provided for the image, the condition specifying a desired style for the image, such that the conditional image is generated to match i) the particular high-level structure captured in the intermediate representation of the image and ii) the desired style specified by the condition provided for the image.

19. The tangible computer readable medium, or media, of claim 18 , wherein:

the first generator stage comprises an unconditional generative neural network, and

the second generator stage comprises a conditional generative neural network.

20. The tangible computer readable medium, or media, of claim 19 , wherein the unconditional generative neural network comprises a structure-prior generative sub-network trained to generate the intermediate representation of the image and an unconditional generative sub-network trained to generate an unconditional image based on the intermediate representation of the image.