Method for few-shot unsupervised image-to-image translation
A few-shot, unsupervised image-to-image translation (“FUNIT”) algorithm is disclosed that accepts as input images of previously-unseen target classes. These target classes are specified at inference time by only a few images, such as a single image or a pair of images, of an object of the target type. A FUNIT network can be trained using a data set containing images of many different object classes, in order to translate images from one class to another class by leveraging few input images of the target class. By learning to extract appearance patterns from the few input images for the translation task, the network learns a generalizable appearance pattern extractor that can be applied to images of unseen classes at translation time for a few-shot image-to-image translation task.
1 . A computer-implemented method, comprising:
using one or more neural networks to generate one or more images of one or more first objects based, at least in part, on one or more pose-independent features of the one or more first objects and one or more poses of one or more second objects,
wherein the one or more neural networks comprise a generative adversarial network (GAN) including a conditional image generator and a multi-task adversarial discriminator.
2 . The computer-implemented method of claim 1 , further comprising:
extracting a class-invariant representation of the one or more poses using a content encoder of the conditional image generator; and
extracting a class-specific latent representation of an appearance of the one or more first objects using a class encoder of the conditional image generator.
3 . The computer-implemented method of claim 1 , further comprising:
normalizing, by a normalization layer of the multi-task adversarial discriminator, layer activations to zero mean and unit variance distribution; and
de-normalizing the normalized layer activations using an affine transformation.
4 . The computer-implemented method of claim 1 , further comprising:
performing adversarial training of the one or more neural networks to translate poses between a plurality of object classes.
5 . The computer-implemented method of claim 1 , comprising:
receiving a digital representation of one or more additional images of the one or more first objects; and
inferring, using the one or more neural networks, the one or more images of the one or more first objects, wherein the conditional image generator and the multi-task adversarial discriminator are to extract a class-invariant representation of the one or more poses and a class-specific representation of an appearance of the one or more first objects.
6 . The computer-implemented method of claim 1 , wherein the one or more neural networks were not trained using images of an object class of the one or more first objects.
7 . The computer-implemented method of claim 1 , further comprising:
performing unsupervised training of the one or more neural networks using a training data set comprising a plurality of images of multiple object classes, wherein the one or more neural networks learn to translate poses between the multiple object classes.
8 . The computer-implemented method of claim 3 , further comprising:
feeding class latent code to an adaptive instance-norm (AdaIN) decoder to decode the class latent code to a set of mean and variance vectors functioning as new means and variances of respective channels in a respective AdaIN residual block of an image decoder of the one or more neural networks.
9 . The computer-implemented method of claim 1 , wherein the conditional image generator includes a content encoder comprising a plurality of first convolutional layers followed by a plurality of residual blocks, and wherein the conditional image generator includes a class encoder comprising a plurality of second convolutional layers followed by an average pooling layer.
10 . The computer-implemented method of claim 9 , further comprising: causing the average pooling layer to average activations across spatial dimensions before averaging activations across a set of images.
11 . A system, comprising:
at least one processor; and
memory including instructions that, if executed by the at least one processor, cause the system to use one or more neural networks to generate one or more images of one or more first objects based, at least in part, on one or more pose-independent features of the one or more first objects and one or more poses of one or more second objects,
wherein the one or more neural networks comprise a generative adversarial network (GAN) including a conditional image generator and a multi-task adversarial discriminator.
12 . The system of claim 11 , wherein the one or more neural networks were not trained using images of an object class of the one or more first objects.
13 . The system of claim 11 , wherein the instructions, if executed by the at least one processor, further cause the system to:
perform unsupervised training of the one or more neural networks using a training data set comprising a plurality of images of different object classes, wherein the one or more neural networks learn to translate poses between the different object classes.
14 . The system of claim 11 , wherein the instructions, if executed by the at least one processor, further cause the system to:
extract a class-invariant representation of the one or more poses using a content encoder of the conditional image generator included in the one or more neural networks; and
extract a class-specific latent representation of an appearance of the one or more first objects using a class encoder of the conditional image generator.
15 . The system of claim 11 , wherein the instructions, if executed by the at least one processor, further cause the system to:
normalize, by a normalization layer of the multi-task adversarial discriminator included in the one or more neural networks, layer activations to zero mean and unit variance distribution;
and de-normalize the normalized layer activations using an affine transformation.
16 . A processor, comprising:
one or more circuits to use one or more neural networks to generate one or more images of one or more first objects based, at least in part, on one or more pose-independent features of the one or more first objects and one or more poses of one or more second objects,
wherein the one or more neural networks comprise a generative adversarial network (GAN) including a conditional image generator and a multi-task adversarial discriminator.
17 . The processor of claim 16 , wherein:
the one or more neural networks were trained using one or more additional images depicting the one or more second objects; and
the one or more neural networks were not trained using images depicting the one or more first objects.
18 . The processor of claim 16 , wherein:
the one or more circuits are to use the one or more neural networks to extract a representation of the one or more poses from one or more images of the one or more second objects; and
the one or more circuits are to use the one or more neural networks to generate the one or more images of the one or more first objects based, at least in part, on the extracted representation of the one or more poses.
19 . The processor of claim 16 , wherein the one or more circuits are to use the one or more neural networks to generate the one or more images of the one or more first objects based, at least in part, on a set of spatially invariant mean and variance vectors generated according to the one or more pose-independent features.
20 . The processor of claim 16 , wherein the one or more circuits are to:
extract a class-invariant representation of the one or more poses using a content encoder of the conditional image generator; and
extract a class-specific latent representation of an appearance of the one or more first objects using a class encoder of the conditional image generator.
21 . The processor of claim 16 , wherein the one or more circuits are to:
normalize, by a normalization layer of the multi-task adversarial discriminator, layer activations to zero mean and unit variance distribution; and
de-normalize the normalized layer activations using an affine transformation.