Context aware text-to-image synthesis
Techniques are generally described for context aware text-to-image synthesis. First text data comprising a description of an object may be received. A recurrent neural network may determine a first semantic representation data representing the first text data. A generator trained using a first generative adversarial network (GAN) may determine first image data representing the object using the first semantic representation. An encoder of a second GAN may generate a first feature representation of the first image data. The first feature representation may be combined with a projection of the first semantic representation data. A decoder of the second GAN may generate second image data representing the first text data.
1. A computer-implemented method of generating synthetic image data, comprising:
receiving first text data describing a first object, wherein the first text data includes a description of a color of the first object;
generating a text embedding of the first text data;
sending the text embedding to an input layer of a long short term memory (LSTM) network;
generating, by the LSTM network, first hidden state data comprising a semantic representation of the first text data;
generating color embedding data representing the color of the first object;
determining a noise vector by sampling a normal distribution of noise values;
combining the first hidden state data, the color embedding data, and the noise vector to generate conditioning data;
sending the conditioning data to an input layer of a first decoder model, wherein the first decoder model is trained as part of a first generative adversarial network (GAN) including the LSTM network; and
generating, by the first decoder model using the conditioning data, first synthetic image data representing the first object, wherein the first synthetic image data comprises a first image quality.
2. The computer-implemented method of claim 1 , further comprising:
sending the first synthetic image data to an encoder model;
generating by the encoder model a feature representation of the first synthetic image data;
generating a projection of the first hidden state data using a fully-connected layer;
concatenating the feature representation of the first synthetic image data with the color embedding data and the projection of the first hidden state data; and
generating second synthetic image data representing the first object by a second decoder model, wherein the second decoder model is trained as part of a second GAN, wherein the second synthetic image data comprises a second image quality that is improved relative to the first image quality.
3. The computer-implemented method of claim 1 , further comprising:
determining a cluster of similar colors included in a color dataset;
determining that the description of the color of the first object in the first text data corresponds to the cluster of similar colors; and
selecting a cluster value from the cluster of similar colors as the color embedding data.
4. A computer-implemented method, comprising;
receiving first text data comprising a description of an object;
determining, by a recurrent neural network, first semantic representation data representing the first text data;
generating, using the first semantic representation data, by a generator of a first generative adversarial network (GAN), first image data representing the object, wherein the recurrent neural network and the first GAN are trained together;
generating, by an encoder of a second GAN, a first feature representation of the first image data;
combining the first feature representation with a projection of the first semantic representation data; and
generating, by a decoder of the second GAN, second image data representing the first text data.
5. The method of claim 4 , further comprising:
receiving second text data comprising a modification of the first text data;
determining, by the recurrent neural network, second semantic representation data representing at least a portion of the first text data and the second text data; and
generating, using the second semantic representation data by the generator, third image data representing the object.
6. The method of claim 4 , further comprising:
determining a noise vector associated with a data distribution; and
combining the first semantic representation data with the noise vector, wherein the noise vector is effective to introduce variability into the first image data generated by the first GAN and the second GAN.
7. The method of claim 4 , further comprising:
determining color embedding data associated with a color indicated by the first text data;
generating conditioning data by combining the first semantic representation data with the color embedding data; and
sending the conditioning data to an input layer of the generator of the first GAN, wherein the color embedding data is used by the generator to select at least one color of the first image data.
8. The method of claim 7 , further comprising combining, by the second GAN, the color embedding data with the first feature representation and the projection of the first semantic representation data, wherein the decoder of the second GAN generates the second image data based at least in part on the color embedding data.
9. The method of claim 4 , further comprising:
generating a first sentence embedding of the first text data at a first time;
sending the first sentence embedding of the first text data to a long short term memory (LSTM) network of the recurrent neural network;
generating, by the LSTM network, first hidden state data representing the first sentence embedding;
generating a second sentence embedding of second text data at a second time, wherein the second text data represents a modification of the first text data;
sending the second sentence embedding of the second text data to the LSTM network; and
generating, by the LSTM network, second hidden state data representing the second sentence embedding and the first sentence embedding, wherein the first semantic representation data comprises the second hidden state data.
10. The method of claim 4 , further comprising:
determining color embedding data associated with a color indicated by the first text data; and
combining, by the second GAN, the color embedding data with the first feature representation and the projection of the first semantic representation data, wherein the generating, by the decoder of the second GAN, the second image data is based at least in part on the color embedding data.
11. The method of claim 4 , further comprising:
generating, by the encoder of the second GAN, the first feature representation in a first number of dimensions;
combining the first semantic representation data with first color embedding data to generate conditioning data, wherein the conditioning data comprises the first number of dimensions; and
concatenating the conditioning data and the first feature representation.
12. The method of claim 4 , further comprising:
generating, by a discriminator of the first GAN, a first label indicating that the first image data is synthetic;
sending a signal to the generator of the first GAN indicating the first label associated with the first image data; and
updating at least one parameter of the generator of the first GAN based at least in part on the signal.
13. A system comprising:
at least one processor; and
at least one non-transitory, computer-readable memory storing instructions that, when executed by the at least one processor, are effective to program the at least one processor to:
receive first text data comprising a description of an object;
determine, by a recurrent neural network, first semantic representation data representing the first text data;
generate, using the first semantic representation data by a generator of a first generative adversarial network (GAN), first image data representing the object, wherein the recurrent neural network and the first GAN are trained together;
generate, by an encoder of a second GAN, a first feature representation of the first image data;
combine the first feature representation with a projection of the first semantic representation data; and
generate, by a decoder of the second GAN, second image data representing the first text data.
14. The system of claim 13 , the at least one non-transitory, computer-readable memory storing further instructions that, when executed by the at least one processor, are effective to further program the at least one processor to:
receive second text data comprising a modification of the first text data;
determine, by the recurrent neural network, second semantic representation data representing at least a portion of the first text data and the second text data; and
generate, using the second semantic representation data by the generator, third image data representing the object.
15. The system of claim 13 , the at least one non-transitory, computer-readable memory storing further instructions that, when executed by the at least one processor, are effective to further program the at least one processor to:
determine a noise vector associated with a data distribution; and
combine the first semantic representation data with the noise vector, wherein the noise vector is effective to introduce variability into the first image data generated by the first GAN and the second GAN.
16. The system of claim 13 , the at least one non-transitory, computer-readable memory storing further instructions that, when executed by the at least one processor, are effective to further program the at least one processor to:
determine color embedding data associated with a color indicated by the first text data;
generate conditioning data by combining the first semantic representation data with the color embedding data; and
send the conditioning data to an input layer of the generator of the first GAN, wherein the color embedding data is used by the generator to select at least one color of the first image data.
17. The system of claim 16 , the at least one non-transitory, computer-readable memory storing further instructions that, when executed by the at least one processor, are effective to further program the at least one processor to combine, by the second GAN, the color embedding data with the first feature representation and the projection of the first semantic representation data, wherein the decoder of the second GAN generates the second image data based at least in part on the color embedding data.
18. The system of claim 13 , the at least one non-transitory, computer-readable memory storing further instructions that, when executed by the at least one processor, are effective to further program the at least one processor to:
generate a first sentence embedding of the first text data at a first time;
send the first sentence embedding of the first text data to a long short term memory (LSTM) network of the recurrent neural network;
generate, by the LSTM network, first hidden state data representing the first sentence embedding;
generate a second sentence embedding of second text data at a second time, wherein the second text data represents a modification of the first text data;
send the second sentence embedding of the second text data to the LSTM network; and
generate, by the LSTM network, second hidden state data representing the second sentence embedding and the first sentence embedding, wherein the first semantic representation data comprises the second hidden state data.
19. The system of claim 13 , the at least one non-transitory, computer-readable memory storing further instructions that, when executed by the at least one processor, are effective to further program the at least one processor to:
determine color embedding data associated with a color indicated by the first text data;
and
combine, by the second GAN, the color embedding data with the first feature representation and the projection of the first semantic representation data, wherein the generating, by the decoder of the second GAN, the second image data is based at least in part on the color embedding data.
20. The system of claim 13 , the at least one non-transitory, computer-readable memory storing further instructions that, when executed by the at least one processor, are effective to further program the at least one processor to:
generate, by the encoder of the second GAN, the first feature representation in a first number of dimensions;
combine the first semantic representation data with first color embedding data to generate condition data, wherein the conditioning data comprises the first number of dimensions; and
concatenate the conditioning data and the first feature representation.