Image processing device comprising face swapping framework, and method therefor
The present invention relates to an image processing device comprising a processor, which uses, when a source image and a target image are input, the source image and the target image so as to generate a face-conversion image, wherein the processor encodes the source image so as to extract an identity feature, encodes the target image so as to extract a target code, decodes the target code so as to extract a pose feature, and integrates the identify feature, the target code and the pose feature, and includes a face swapping framework, which uses an attribute image in which the size of the target image has been adjusted, so as to generate a face-conversion image in which an attribute feature of the target image is reflected.
1 . An image processing device comprising:
a processor configured to execute a face swapping framework;
a memory configured to store the face swapping framework,
wherein the face swapping framework comprises:
an identity encoder configured to encode a source image so as to extract an identity feature;
a pose network configured to encode a target image so as to extract a target code and to decode the target code so as to extract a pose feature; and
a triple adaptive normalization (TAN) decoder configured to integrate the identify feature, the target code and the pose feature and to use an attribute image in which a size of the target image has been adjusted so as to generate a face-conversion image in which an attribute feature of the target image is reflected,
wherein the TAN decoder comprises a plurality of TAN blocks,
wherein each of the plurality of TAN blocks comprises:
an identity integration block configured to produce a result of an identity activation function corresponding to the identity feature;
a spatial-adaptive pose integration block configured to produce a result of a pose activation function corresponding to the pose feature; and
a non-spatial-adaptive pose integration block configured to produce a result of a code activation function corresponding to the target code.
2 . The image processing device of claim 1 , wherein the TAN block comprises a spatial-adaptive branch and a non-spatial-adaptive branch in a parallel form, wherein the spatial-adaptive pose integration block and the identity integration block are arranged in series in sequence in the spatial-adaptive branch, and the non-spatial-adaptive pose integration block is arranged in the non-spatial-adaptive branch.
3 . The image processing device of claim 1 , wherein the face swapping framework performs data augmentation by color distortion on the source image and the target image in a training phase of the face swapping framework.
4 . The image processing device of claim 1 , wherein in a training phase of the face swapping framework, the source image having an adjusted size is input as the attribute image, and
in a testing phase of the face swapping framework, the target image having an adjusted size is input as the attribute image.