Image processing method and apparatus
View Patent ↗Disclosed is an image processing method and apparatus. The image processing method includes extracting a content latent code and a style latent code for each of a plurality of input images, obtaining a content feature vector by calculating a weighted sum of content latent codes extracted from the input images based on a morphing control parameter, obtaining a style feature vector by calculating a weighted sum of style latent codes extracted from the input images based on the morphing control parameter, and generating a morphing image based on the content feature vector and the style feature vector.
1. A training method, comprising:
obtaining training data comprising a first image, a second image, and a ground truth morphing image that is based on a morphing control parameter, wherein the ground truth morphing image is generated by linearly interpolating a first point corresponding to the first image sampled in a latent space by a pretrained model and a second point corresponding to the second image sampled in the latent space by the pretrained model, based on the morphing control parameter;
obtaining an output morphing image by inputting the first image, the second image, and the morphing control parameter to a morphing generator; and
training the morphing generator based on a loss function that is based on at least one of the images comprised in the training data and the output morphing image,
wherein the loss function is determined based on at least one of:
an adversarial loss associated with a discrimination between the image comprised in the training data and the output morphing image obtained from the morphing generator; or
a pixel-wise reconstruction loss associated with a difference between pixels of the image comprised in the training data and pixels of the output morphing image.
2. The training method of claim 1 , wherein the adversarial loss comprises a conditional adversarial loss in which class label information of one of the images comprised in the training data is set as a condition.
3. The training method of claim 1 , wherein the adversarial loss is determined based on the morphing control parameter, an adversarial loss associated with a discrimination between the first image and the output morphing image, and an adversarial loss associated with a discrimination between the second image and the output morphing image, and
the pixel-wise reconstruction loss is determined based on the ground truth morphing image corresponding to the morphing control parameter among the images comprised in the training data and on the output morphing image.
4. The training method of claim 1 , wherein the morphing control parameter comprises a content morphing control parameter and a style morphing control parameter,
wherein the obtaining of the output morphing image comprises:
obtaining a first output morphing image comprising a content of the second image and a style of the first image; and
obtaining a second output morphing image comprising a content of the first image and a style of the second image,
wherein the training of the morphing generator comprises:
training the morphing generator based on an adversarial loss associated with a discrimination between the first output morphing image and the first image, and an adversarial loss associated with a discrimination between the second output morphing image and the second image.
5. The training method of claim 4 , wherein the training of the morphing generator comprises:
obtaining a third morphing image comprising a content of the first output morphing image and the style of the second image by inputting the first output morphing image and the second image to the morphing generator;
obtaining a fourth morphing image comprising a content of the second output morphing image and the style of the first image by inputting the second output morphing image and the first image to the morphing generator;
training the morphing generator based on an adversarial loss and a pixel-wise reconstruction loss that are based on the third output morphing image and the second image; and
training the morphing generator based on an adversarial loss and a pixel-wise reconstruction loss that are based on the fourth output morphing image and the first image.
6. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the image processing method of claim 1 .
7. An image processing apparatus, comprising:
at least one processor;
wherein, the processor is configured to:
obtain training data comprising a first image, a second image, and a ground truth morphing image that is based on a morphing control parameter, wherein the ground truth morphing image is generated by linearly interpolating a first point corresponding to the first image sampled in a latent space by a pretrained model and a second point corresponding to the second image sampled in the latent space by the pretrained model, based on the morphing control parameter;
obtain an output morphing image by inputting the first image, the second image, and the morphing control parameter to a morphing generator; and
train the morphing generator based on a loss function that is based on at least one of the images comprised in the training data and the output morphing image,
wherein the loss function is determined based on at least one of:
an adversarial loss associated with a discrimination between the image comprised in the training data and the output morphing image obtained from the morphing generator; or
a pixel-wise reconstruction loss associated with a difference between pixels of the image comprised in the training data and pixels of the output morphing image.
8. The image processing apparatus of claim 7 , wherein the adversarial loss comprises a conditional adversarial loss in which class label information of one of the images comprised in the training data is set as a condition.
9. The image processing apparatus of claim 7 , wherein the adversarial loss is determined based on the morphing control parameter, an adversarial loss associated with a discrimination between the first image and the output morphing image, and an adversarial loss associated with a discrimination between the second image and the output morphing image, and
the pixel-wise reconstruction loss is determined based on the ground truth morphing image corresponding to the morphing control parameter among the images comprised in the training data and on the output morphing image.
10. The image processing apparatus of claim 7 , wherein the morphing control parameter comprises a content morphing control parameter and a style morphing control parameter,
for the obtaining of the output morphing image, wherein the processor is configured to:
obtain a first output morphing image comprising a content of the second image and a style of the first image; and
obtain a second output morphing image comprising a content of the first image and a style of the second image,
wherein the training of the morphing generator comprises:
training the morphing generator based on an adversarial loss associated with a discrimination between the first output morphing image and the first image, and an adversarial loss associated with a discrimination between the second output morphing image and the second image.
11. The image processing apparatus of claim 10 , for the training of the morphing generator, wherein the processor is configured to:
obtain a third morphing image comprising a content of the first output morphing image and the style of the second image by inputting the first output morphing image and the second image to the morphing generator;
obtain a fourth morphing image comprising a content of the second output morphing image and the style of the first image by inputting the second output morphing image and the first image to the morphing generator;
train the morphing generator based on an adversarial loss and a pixel-wise reconstruction loss that are based on the third output morphing image and the second image; and
train the morphing generator based on an adversarial loss and a pixel-wise reconstruction loss that are based on the fourth output morphing image and the first image.