METHOD AND SYSTEM FOR TRAINING A MODEL FOR IMAGE GENERATION
A method and system for training a model for image generation. The model includes a hybrid variational auto-encoder (VAE)—generative adversarial network (GAN) framework. The method includes the steps of: multiple input of an input image into the VAE which outputs in response multiple distinct output image samples, determining the best of the multiple output image samples as a best-of-many sample, the best-of-many sample having the minimum reconstruction cost, and training the model based on a predefined training objective, the predefined training objective integrating the best-of-many sample reconstruction cost and a GAN-based synthetic likelihood term.
1 .- 15 . (canceled)
16 . A method of training a model for image generation,
the model comprising a hybrid variational auto-encoder (VAE)—generative adversarial network (GAN) framework,
the method comprising the steps of:
a—multiple input of an input image into the VAE which outputs in response multiple distinct output image samples,
b—determine the best of the multiple output image samples as a best-of-many sample, the best-of-many sample having the minimum reconstruction cost,
c—train the model based on a predefined training objective, the predefined training objective integrating the best-of-many sample reconstruction cost and a GAN-based synthetic likelihood term.
17 . The method according to claim 16 , wherein
the model is trained by using only the best-of-many sample for training the model and by disregarding the further multiple output image samples.
18 . The method according to claim 16 , wherein
the model is trained based on the best-of-many sample in relation to the input image according to a predefined VAE objective.
19 . The method according to claim 16 , wherein
the model is a deep neural network or comprises at least one deep neural network.
20 . The method according to claim 16 , wherein
the model comprises:
a variational auto-encoder (VAE) including a recognition network and a generator, and
a generative adversarial network (GAN) including a generator and a discriminator.
21 . The method according to claim 20 , wherein
the variational auto-encoder (VAE) and the generative adversarial network (GAN) share a common generator.
22 . The method according to claim 16 , wherein
the model is trained in step c based on the GAN-based synthetic likelihood term to learn generating sharper images by leveraging a discriminator of the GAN which is jointly trained to distinguish between real and generated images.
23 . The method according to claim 22 , wherein
during each training iteration the latent distribution of the input image is sampled by:
multiple input of the input image into a recognition network which outputs in response respective regions in a latent space, and
generation of respective output image samples in the image space by inputting the respective regions in the latent space into a generator.
24 . The method according to claim 16 , wherein
the output image samples are inputted into a discriminator of the GAN which outputs the GAN-based synthetic likelihood term.
25 . The method according to claim 16 , wherein
only the worst of the multiple output image samples is inputted into a discriminator of the GAN which outputs the GAN-based synthetic likelihood term.
26 . The method according to claim 16 , wherein
the Lipschitz constant of the GAN-based synthetic likelihood term is constrained to be equal to a predetermined value using Spectral Normalization.
27 . The method according to claim 26 , wherein the predetermined value is equal to 1.
28 . A system for training a model for image generation,
the model comprising a hybrid variational auto-encoder (VAE)—generative adversarial network (GAN) framework, the system comprising:
a module A configured for a multiple input of an input image into the VAE which outputs in response multiple distinct output image samples,
a module B for determining the best of the multiple output image samples as a best-of-many sample, the best-of-many sample having the minimum reconstruction cost, and
a module C for training the model based on a predefined training objective, the predefined training objective integrating the best-of-many sample reconstruction cost and a GAN-based synthetic likelihood term.
29 . The system according to claim 28 , further comprising the model.
30 . A system for generating an image sample,
comprising one of the trained model of step c of claim 16 and the trained module C of claim 16 , wherein
the Lipschitz constant of the GAN-based synthetic likelihood term is constrained to be equal to a predetermined value using Spectral Normalization.
31 . A computer program comprising instructions for executing the steps of the method according to claim 16 , when the program is executed by a computer.
32 . A recording medium readable by a computer and having recorded thereon a computer program including instructions for executing the steps of a method according to claim 16 .