Cascaded domain bridging for image generation
A method of generating a style image is described. The method includes receiving an input image of a subject. The method further includes encoding the input image using a first encoder of a generative adversarial network (GAN) to obtain a first latent code. The method further includes decoding the first latent code using a first decoder of the GAN to obtain a normalized style image of the subject, wherein the GAN is trained using a loss function according to semantic regions of the input image and the normalized style image.
1. A method of generating a style image, the method comprising:
receiving an input image of a subject;
encoding the input image using a first encoder of a generative adversarial network (GAN) to obtain a first latent code;
decoding the first latent code using a first decoder of the GAN to obtain a normalized style image of the subject, wherein:
the GAN is trained using a loss function according to semantic regions of the input image and the normalized style image, and
a distribution prior of a W+ space is modeled for training the GAN by inverting a dataset of real face images using a second encoder that is pre-trained.
2. The method of claim 1 , further comprising training the GAN by inverting the dataset of real face images to obtain a plurality of latent codes.
3. The method of claim 1 , the second encoder is different from the first decoder.
4. The method of claim 1 , wherein the second encoder is a pre-trained StyleGAN encoder.
5. The method of claim 1 , wherein training the GAN further comprises performing a W+ space transfer learning from the second encoder to the first encoder.
6. The method of claim 5 , wherein performing the W+ space transfer learning comprises using a normalized exemplar set with only neutral expressions of the subject.
7. The method of claim 5 , wherein performing the W+ space transfer learning comprises using a normalized exemplar set with only neutral poses of the subject.
8. The method of claim 5 , wherein performing the W+ space transfer learning comprises using a normalized exemplar set with only neutral lighting of the subject.
9. The method of claim 1 , further comprising training the GAN using a difference between a first face segmentation model trained using real face images and a second face segmentation model using style exemplars as the loss function.
10. The method of claim 9 , wherein the semantic regions include one or more of hair regions of the subject or skin regions of the subject.
11. A system for generating a style image, the system comprising:
a processor; and
memory storing instructions that, when executed by the processor, cause the system to perform a set of operations, the set of operations comprising:
receiving an input image of a subject;
encoding the input image using a first encoder of a generative adversarial network (GAN) to obtain a first latent code;
decoding the first latent code using a first decoder of the GAN to obtain a normalized style image of the subject, wherein:
the GAN is trained using a loss function according to semantic regions of the input image and the normalized style image, and
a distribution prior of a W+ space is modeled for training the GAN by inverting a dataset of real face images using a second encoder that is pre-trained.
12. The method of claim 11 , wherein the set of operations further comprise training the GAN by inverting the dataset of real face images to obtain a plurality of latent codes.
13. The method of claim 11 , wherein the second encoder is different from the first decoder.
14. The method of claim 11 , wherein the second encoder is a pre-trained StyleGAN encoder.
15. The method of claim 11 , wherein the set of operations further comprise performing a W+ space transfer learning from the second encoder to the first encoder.
16. The method of claim 15 , wherein the set of operations further comprise using a normalized exemplar set with only neutral expressions of the subject.
17. The method of claim 15 , wherein the set of operations further comprise using a normalized exemplar set with only neutral poses of the subject.
18. The method of claim 15 , wherein the set of operations further comprise using a normalized exemplar set with only neutral lighting of the subject.
19. The method of claim 11 , wherein the set of operations further comprise training the GAN using a difference between a first face segmentation model trained using real face images and a second face segmentation model using style exemplars as the loss function.
20. The method of claim 19 , wherein the semantic regions include one or more of hair regions of the subject or skin regions of the subject.