Unsupervised visual attribute transfer through reconfigurable image translation
The present disclosure relates to unsupervised visual attribute transfer through reconfigurable image translation. One aspect of the present disclosure provides a system for learning the transfer of visual attributes, including an encoder, converter and generator. The encoder encodes an original source image to generate a plurality of attribute values that specify the original source image, and to encode an original reference image to generate a plurality of attribute values that specify the original reference image. The converter replaces at least one attribute value of an attribute that is target attribute of the attribute values of the original source image with at least one corresponding attribute value of the original reference image, to obtain a plurality of attribute values that specify a target image of interest. The generator generates a target image based on the attribute values of the target image of interest.
1. A system for learning an attribute transfer that conveys at least one attribute value of a reference image to a source image, the system comprising:
an encoder configured to encode an original source image to generate a plurality of attribute values that specify the original source image, and to encode an original reference image to generate a plurality of attribute values that specify the original reference image;
a converter configured to replace at least one attribute value of a target attribute of the attribute values of the original source image with at least one corresponding attribute value of the original reference image, to obtain a plurality of attribute values that specify a target image of interest; and
a generator configured to generate a target image based on the attribute values of the target image of interest,
wherein during training the system,
the encoder encodes the generated target image to generate a plurality of attribute values that specify the generated target image,
the converter replaces at least one attribute value corresponding to the target attribute among the attribute values of the generated target image with at least one attribute value corresponding to the target attribute of the original source image, to generate a plurality of attribute values for a source image reconstruction,
the generator generates a reconstructed source image based on the attribute values of the source image reconstruction,
the converter replaces at least one attribute value corresponding to the target attribute among the attribute values of the original reference image with at least one attribute value corresponding to the target attribute of the target image, to generate a plurality of attribute values for a reference image reconstruction, and
the generator generates a reconstructed reference image based on the attribute values for the reference image reconstruction,
wherein, during training the system, parameters of the encoder and the generator are updated by using,
a reconstruction loss that represents a difference between the reconstructed source image and the original source image,
a reconstruction loss that represents a difference between the reconstructed reference image and the original reference image, and
a generative adversarial loss of the generated target image.
2. The system of claim 1 , further comprising a discriminator configured to learn a discrimination model for discriminating a generated target image from the original source image.
3. A computer-implemented method of training one or more artificial neural networks to learn a visual attribute transfer that conveys a target attribute of a reference image to a source image, for causing the artificial neural networks to perform the computer-implemented method comprising:
encoding an original source image to generate a plurality of attribute values that specify the original source image, and encoding an original reference image to generate a plurality of attribute values that specify the original reference image;
in order to generate a plurality of attribute values that specify a target image of interest, replacing at least one attribute value of a target attribute from among the attribute values of the original source image with at least one corresponding attribute value of the original reference image;
generating a target image based on the attribute values of the target image of interest;
encoding the generated target image to generate a plurality of attribute values that specify a generated target image;
in order to generate a plurality of attribute values for a source image reconstruction, replacing at least one attribute value corresponding to the target attribute from among attribute values of the generated target image with at least one attribute value corresponding to the target attribute of the original source image;
in order to generate a plurality of attribute values for a reference image reconstruction, replacing at least one attribute value corresponding to the target attribute from among the attribute values of the original reference image with at least one attribute value corresponding to the target attribute of the target image;
generating a reconstructed source image based on the attribute values for the source image reconstruction;
generating a reconstructed reference image based on the attribute values for the reference image reconstruction; and
updating parameters of the artificial neural networks by using:
a reconstruction loss that represents a difference between the reconstructed source image and the original source image,
a reconstruction loss that represents a difference between the reconstructed reference image and the original reference image, and
a generative adversarial loss of the generated target image.