Method and device for generative adversarial network training

1. An electronic device for neural network training comprising:

one or more processors;

a non-transitory computer-readable medium storing one or more programs and data representative of a generative adversarial network (GAN) having a plurality of nodes and weights, the GAN including a generator and a discriminator, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs including instructions for:

receiving data representative of a plurality of training samples;

generating, by the generator, a plurality of hard negative samples based on the plurality of training samples, wherein the hard negative samples are generated from: samples which are initially misclassified by the GAN, or samples which are classified differently than statistically-similar samples, wherein the generator is based on a fixed noise distribution and a conditional distribution defining a categorical distribution over possible training samples; and

generating an output, by the discriminator, based on the plurality of hard negative samples from the generator.

2. The electronic device of claim 1 , wherein at least one of the generator and the discriminator includes a neural network.

3. The electronic device of claim 2 , wherein the generator is defined for an image or caption retrieval.

4. The electronic device of claim 3 , wherein the generator for the image retrieval is based on:

λ p noise ( i )+(1−λ) g θ ( i|c )

wherein p noise is a fixed noise distribution, g θ is a conditional distribution with learnable parameters θ and λ is a hyperparameter, g θ (i|c) and defines a categorical distribution over all possible images from a set of images in the plurality of training cases.

5. The electronic device of claim 3 , wherein the generator for the caption retrieval is based on:

λ p noise ( c )+(1−λ) g θ ( c|i )

wherein p noise is a fixed noise distribution, g θ is a conditional distribution with learnable parameters θ and λ is a hyperparameter, and g θ (i|c) defines a categorical distribution over all possible images from a set of images including correctly labelled images.

6. The electronic device of claim 2 , wherein an output of the discriminator is used as a feedback to the generator.

7. The electronic device of claim 3 , wherein the a score for an image-caption pair is used as a reward for the generator.

8. The electronic device of claim 7 , wherein the GAN is configured to: receive data representative of an image, and generate an output to identify, from a plurality of text strings, a corresponding text string based on the data representative of the image.

9. The electronic device of claim 7 , wherein the GAN is configured to: receive data representative of a text string, and generate an output to identify, from a plurality of images, a corresponding image based on the data representative of the text string.

10. A computer-implemented method comprising:

receiving, by a generative adversarial network (GAN) having a plurality of nodes and weights, the GAN including a generator and a discriminator, data representative of a plurality of training samples;

generating, by the generator, a plurality of hard negative samples based on the plurality of training samples, wherein the hard negative samples are generated from: samples which are initially misclassified by the GAN, or samples which are classified differently than statistically-similar samples, wherein the generator is based on a fixed noise distribution and a conditional distribution defining a categorical distribution over possible training samples; and

generating an output, by the discriminator, based on the plurality of hard negative samples from the generator.

11. The method of claim 10 , wherein at least one of the generator and the discriminator includes a neural network.

12. The method of claim 11 , wherein the generator for the image retrieval is based on:

λ p noise ( i )+(1−λ) g θ ( i|c )

wherein p noise is a fixed noise distribution, g θ is a conditional distribution with learnable parameters θ and λ is a hyperparameter, and g θ (i|c) defines a categorical distribution over all possible images from a set of images in the plurality of training cases.

13. The method of claim 11 , wherein the generator for the caption retrieval is based on:

λ p noise ( c )+(1−λ) g θ ( c|i )

wherein p noise is a fixed noise distribution, g θ is a conditional distribution with learnable parameters θ and λ is a hyperparameter, g θ (i|c) and defines a categorical distribution over all possible images from a set of images including correctly labelled images.

14. The method of claim 10 , wherein an output of the discriminator is used as a feedback to the generator.

15. The method of claim 11 , wherein the a score for an image-caption pair is used as a reward for the generator.

16. The method of claim 15 , wherein the GAN is configured to: receive data representative of an image, and generate an output to identify, from a plurality of text strings, a corresponding text string based on the data representative of the image.

17. The method of claim 15 , wherein the GAN is configured to: receive data representative of a text string, and generate an output to identify, from a plurality of images, a corresponding image based on the data representative of the text string.

18. An electronic device comprising:

one or more processors; and

a memory storing one or more program configured to be executed by the one or more processors, the one or more programs including instructions for:

receiving a text string or an image;

processing the text string or the image using a generative adversarial network including a generator and a discriminator; and

choosing an matched image based on the processed text string from a plurality of images, or choosing a matched text string based on the processed image from a plurality of text strings;

wherein during a training phase, the generator is configured to generate a plurality of hard negatives samples for the discriminator, wherein the hard negative samples are generated from: samples which are initially misclassified by the GAN, or samples which are classified differently than statistically-similar samples, wherein the generator is based on a fixed noise distribution and a conditional distribution defining a categorical distribution over possible training samples.

19. The method of claim 18 , wherein the generator is based on:

λ p noise ( i )+(1−λ) g θ ( i|c )

wherein p noise is a fixed noise distribution, g θ is a conditional distribution with learnable parameters θ and λ is a hyperparameter, and g θ (i|c) defines a categorical distribution over all possible images from a set of images in a plurality of training cases.

20. The method of claim 18 , wherein the generator for is based on:

λ p noise ( c )+(1−λ) g θ ( c|i )

wherein p noise is a fixed noise distribution, g θ is a conditional distribution with learnable parameters θ and λ is a hyperparameter, g θ (i|c) and defines a categorical distribution over all possible images from a set of images including correctly labelled images.