Optimizing supervised generative adversarial networks via latent space regularizations

1. A method of training a generator G of a Generative Adversarial Network (GAN), comprising:

receiving, by an encoder E, a target data Y;

receiving, by the encoder E, an output G(Z) of the generator G,

wherein the generator G generates the output G(Z) in response to receiving a random sample Z that is a noisy sample, and

wherein a discriminator D of the GAN is trained to distinguish which of the G(Z) and the target data Y is real data;

training the encoder E to minimize a difference between a first latent space representation E(G(Z)) of the output G(Z) and a second latent space representation E(Y) of the target data Y, wherein the output G(Z) and the target data Y are input to the encoder E; and

using the first latent space representation E(G(Z)) and the second latent space representation E(Y) to constrain the training of the generator G.

2. The method of claim 1 , further comprising:

updating the encoder E by descending, for m samples, a gradient

∇

θ

E

⁢

1

m

⁢

∑

i

=

1

m

⁢



E

θ

E

⁡

(

y

i

)

-

E

θ

E

⁡

(

G

θ

G

⁡

(

z

i

)

)



,

wherein the m samples comprising m noise samples {z 1 , z 2 , . . . , z m } selected from a distribution p(z) and m corresponding target data {y 1 , y 2 , . . . , y m },

wherein E θ E (y i ) is a first output of the encoder E given first current weight values θ E of the encoder E and a first input y i ,

wherein G θ G (z i ) is a second output of the of the generator G given second current weight values θ G of the generator G and a second input z i , and

wherein E θ E (G θ G (z i )) is a third output of the of the encoder E given the first current weight values θ E of the encoder E and G θ G (z i ) as input.

3. The method of claim 1 , wherein the target data Y corresponds to the output G(Z).

4. The method of claim 1 , wherein the generator G is trained by applying a Lipschitz condition so as to upper bound a first difference between the output G(Z) and the target data Y to a second difference between the first latent space representation E(G(Z)) and the second latent space representation E(Y).

5. The method of claim 1 ,

wherein the encoder E is a neural network comprising an upscaling layer, and

wherein each of the outputs E(G(Z)) and E(Y) of the encoder E having a first size equal to a second size of Z.

6. The method of claim 1 , wherein the random sample Z is a multi-dimensional vector of values selected from a distribution p(z).

7. The method of claim 1 , wherein using the first latent space representation E(G(Z)) and the second latent space representation E(Y) to constrain the training of the generator G, comprising:

updating the generator G by descending, for in samples, a gradient

∇

θ

G

⁢

1

m

⁢

∑

i

=

1

m

⁢

[

log

⁢

(

1

-

D

θ

D

⁡

(

G

θ

G

⁡

(

z

i

)

)

)

+

μ

1

*



y

i

-

G

θ

G

⁡

(

z

i

)



-

μ

2

*



E

θ

E

⁡

(

y

i

)

-

E

θ

E

⁡

(

G

θ

G

⁡

(

z

i

)

)



]

wherein θ D , θ G , and θ E are, respectively, weights of the discriminator D, the generator G, and the encoder E,

wherein G θ G (z i ) is an output of the generator G for an input z i ,

wherein D θ D (G θ D (z i )) and E θ E (G θ G (z i )) are, respectively, outputs of the discriminator D and the encoder E when G θ G (z i ) is used as input,

wherein E θ E (y i ) is an encoder E output for a target datum y i , and

wherein μ 1 and μ 2 are hyperparameters.

8. An apparatus for training a generator G of a Generative Adversarial Network (GAN), comprising:

a memory; and

a processor, the processor is configured to execute instructions stored in the memory to:

input, to an encoder E, a target data Y;

input, to the encoder E, an output G(Z) of the generator G,

wherein the generator G generates the output G(Z) in response to receiving a random sample Z, and

wherein a discriminator D of the GAN is trained to distinguish which of the G(Z) and the target data Y is real data;

train the encoder E to minimize a difference between a first latent space representation E(G(Z)) of the output G(Z) and a second latent space representation E(Y) of the target data Y, wherein the output G(Z) and the target data Y are input to the encoder E; and

use the first latent space representation E(G(Z)) and the second latent space representation E(Y) to constrain the training of the generator G.

9. The apparatus of claim 8 , wherein the instructions further comprise instructions to:

update the encoder E by descending, for m samples, a gradient

∇

θ

E

⁢

1

m

⁢

∑

i

=

1

m

⁢



E

θ

E

⁡

(

y

i

)

-

E

θ

E

⁡

(

G

θ

G

⁡

(

z

i

)

)



,

wherein the m samples comprise m noise samples {z 1 , z 2 , . . . , z m } selected from a distribution p(z) and m corresponding target data {y 1 , y 2 , . . . , y m },

wherein E θ E (y i ) is a first output of the encoder E given first current weight values θ E of the encoder E and a first input y i ,

wherein G θ G (z i ) is a second output of the of the generator G given second current weight values θ G of the generator G and a second input z i , and

wherein E θ E (G θ G (z i )) is a third output of the of the encoder E given the first current weight values θ E of the encoder E and G θ G (z j ) as input.

10. The apparatus of claim 8 , wherein the target data Y corresponds to the output G(Z).

11. The apparatus of claim 8 , wherein the encoder E comprises a neural network.

12. The apparatus of claim 8 , wherein the random sample Z is a multi-dimensional vector of values selected from a distribution p(z).

13. The apparatus of claim 8 , wherein to use the first latent space representation E(G(Z)) and the second latent space representation E(Y) to constrain the training of the generator G, comprises to:

updating the generator G by descending, form samples, a gradient

∇

θ

G

⁢

1

m

⁢

∑

i

=

1

m

⁢

[

log

⁢

(

1

-

D

θ

D

⁡

(

G

θ

G

⁡

(

z

i

)

)

)

+

μ

1

*



y

i

-

G

θ

G

⁡

(

z

i

)



-

μ

2

*



E

θ

E

⁡

(

y

i

)

-

E

θ

E

⁡

(

G

θ

G

⁡

(

z

i

)

)



]

.

14. The apparatus of claim 11 ,

wherein the encoder E further comprises an upscaling layer, and

wherein each of the outputs E(G(Z)) and E(Y) of the encoder E having a first size equal to a second size of Z.

15. A method for generating a super resolution image, comprising:

receiving, by a generator G, an input corresponding to a low resolution image; and

outputting, from the generator G, a super resolution image corresponding to the low resolution image, wherein the generator is trained using a Generative Adversarial Network (GAN) comprising:

the generator G;

an encoder E; and

a discriminator D, wherein outputs of the encoder E are used to constrain the training of the generator G, and wherein the outputs of the encoder E comprise a first latent space representation E(G(Z)) of and output G(Z) of the generator G, wherein Z corresponds to a training low-resolution image and G(Z) corresponds to a generated high-resolution image.

16. The method of claim 15 , wherein the outputs of the encoder E further comprises:

a second latent space representation E(Y) of a training high-resolution image Y.

17. The method of claim 15 , wherein the encoder E comprises a VGG network.

18. The method of claim 16 , wherein the encoder E is trained to minimize a difference between the first latent space representation E(G(Z)) and the second latent space representation E(Y).

19. The method of claim 18 , wherein the encoder E is trained by descending, for m samples, a gradient

∇

θ

E

⁢

1

m

⁢

∑

i

=

1

m

⁢



E

θ

E

⁡

(

y

i

)

-

E

θ

E

⁡

(

G

θ

G

⁡

(

z

i

)

)



.

20. The method of claim 18 , wherein the generator G is trained by applying a Lipschitz condition so as to upper bound a first difference between the generated high-resolution image and the training high-resolution image Y to a second difference between the first latent space representation E(G(Z)) and the second latent space representation E(Y).