Method and apparatus for transforming input image based on target style and target color information
An image transformation method includes receiving an input image, a target style and target color information; generating a texture-transformed image in which the input image is transformed to a texture corresponding to the target style and a color-transformed image in which the input image is transformed to a color corresponding to the target color information, using a pretrained transformation model; and synthesizing the texture-transformed image and the color-transformed image to generate a result image in which the input image is transformed according to the target style and the target color information.
1 . An image transformation method performed by a computer system, the image transformation method comprising:
receiving, from a user, an input image, a target style to which the input image is to be transformed, and target color information to which the input image is to be transformed;
generating, responsive to the receipt of the input image, the target style and the target color information from the user, a texture-transformed image in which the input image is transformed to a texture corresponding to the target style and a color-transformed image in which the input image is transformed to a color corresponding to the target color information, using a pretrained transformation model; and
synthesizing the texture-transformed image and the color-transformed image to generate a result image in which the input image is transformed according to the target style and the target color information,
wherein the generating of the texture-transformed image and the color-transformed image respectively includes,
using a texture decoder of the transformation model to generate the texture-transformed image based on the input image in which a texture corresponding to the target style is applied; and
using a color decoder of the transformation model to generate the color-transformed image based on the input image in which a color corresponding to the target color is applied.
2 . The image transformation method of claim 1 , wherein the transformation model is pretrained to transform the input image to represent the texture corresponding to the target style and the color corresponding to the input target color information without receiving an input of a reference image that represents the target style.
3 . The image transformation method of claim 1 , wherein the target color information includes a palette including a group of a plurality of different colors determined based on a color distribution of the input image.
4 . The image transformation method of claim 3 , wherein each of the plurality of different colors is configured to be selectable by a user.
5 . The image transformation method of claim 1 , wherein the target style represents content that includes an image or a video, and
the texture corresponding to the target style is a texture of the image or the video included in the content.
6 . The image transformation method of claim 1 , wherein the receiving of the input image, a target style and target color information comprises receiving, from the user, an abstraction level for setting a degree of abstraction to be represented by the result image, and
the texture-transformed image is generated by transforming the input image to the texture corresponding to the target style that reflects the degree of abstraction represented by the abstraction level.
7 . The image transformation method of claim 1 , further comprising:
an operation of preprocessing the input image including,
transforming the input image to a first transformation image that is an image of a Lab color space;
generating a first color map by performing simplification processing on the input image;
generating a second color map by transforming a color of the first color map based on the target color information; and
transforming the second color map to a second transformation image that is an image of the lab color space,
wherein the first transformation image and the second transformation image are input to the transformation model, and
the texture-transformed image is generated based on the first transformation image and the color-transformed image is generated based on the second transformation image.
8 . The image transformation method of claim 7 , wherein the generating of the texture-transformed image and the color-transformed image by the transformation model includes,
encoding the first transformation image using an encoder;
generating the texture-transformed image by decoding the encoded first transformation image using the texture decoder; and
generating the color-transformed image by decoding the encoded first transformation image based on the second transformation image using the color decoder.
9 . The image transformation method of claim 1 , wherein the texture-transformed image is an image of an L component as an image of a Lab color space, and
the color-transformed image is an image of an ab component as the image of the Lab color space, and
the synthesizing of the texture-transformed image and the color-transformed image the comprises:
generating a composite image by synthesizing the image of the L component and the image of the ab component; and
transforming the composite image to an image of an RGB color space.
10 . The image transformation method of claim 6 , wherein the transformation model comprises a degree-of-abstraction module configured with a plurality of layers, each layer generating an output value of reflecting a degree of abstraction represented by each abstraction level, and
the texture-transformed image is generated by transforming the input image to the texture corresponding to the target style that reflects the degree of abstraction represented by the received abstraction level, using an output value of a layer of the degree-of-abstraction module corresponding to the received abstraction level.
11 . The image transformation method of claim 1 , wherein the transformation model is pretrained using first training data generated by performing data augmentation on at least one first image that includes the texture corresponding to the target style and second training data generated by performing data augmentation on at least one second image that is a target image for transformation to the target style and color transformation.
12 . The image transformation method of claim 11 , wherein the first training data includes an image on which at least one of resizing processing and resolution change processing is performed, to simulate a desired degree of abstraction processing for the first image.
13 . The image transformation method of claim 11 , wherein the second training data includes a transformation image for training in which the second image is color-transformed to an arbitrary color.
14 . The image transformation method of claim 13 , wherein the transformation image is generated by transformation operations, and
the transformation operations comprise:
an operation of transforming at least one of the second image and a color map of the second image generated by performing simplification processing on the second image to an image of a hue saturation value (HSV) color space; and
an operation of color-transforming the image of the HSV color space to an arbitrary color.
15 . The image transformation method of claim 14 , wherein the transformation operations further comprise:
an operation of transforming at least one of the second image and the color map of the second image to an image of a Lab color image;
an operation of extracting L information of the image of the Lab color image;
an operation of transforming the image of the HSV color space that is color-transformed to the arbitrary color to the image of the Lab color image; and
an operation of substituting L information of the transformed image of the Lab color space with the extracted L information.
16 . The image transformation method of claim 13 , wherein the transformation model is pretrained using an image acquired by transforming the first training data to an L component of an image of a Lab color space, an image acquired by transforming the second image to the image of the Lab color space, an image acquired by transforming an image in which the second image is color-transformed to the arbitrary color to an image of an ab component of the Lab color space, and an image acquired by transforming a color map of the second image to the image of the Lab color space.
17 . A non-transitory computer-readable recording medium storing instructions that, when executed by a processor, cause the processor to perform the image transformation method of claim 1 on the computer system.
18 . A computer system to perform an image transformation method, the computer system comprising:
at least one processor configured to execute instructions readable in the computer system,
wherein the at least one processor is configured to:
receive, from a user, an input image, a target style to which the input image is to be transformed, and target color information to which the input image is to be transformed,
generate, responsive to the receipt of the input image, the target style and the target color information from the user, a texture-transformed image in which the input image is transformed to a texture corresponding to the target style and a color-transformed image in which the input image is transformed to color corresponding to the target color information, using a pretrained transformation model, and
synthesize the texture-transformed image and the color-transformed image to generate a result image in which the input image is transformed according to the target style and the target color information,
wherein the generating of the texture-transformed image and the color-transformed image respectively includes,
using a texture decoder of the transformation model to generate the texture-transformed image based on the input image in which a texture corresponding to the target style is applied; and
using a color decoder of the transformation model to generate the color-transformed image based on the input image in which a color corresponding to the target color is applied.
19 . A method of training a transformation model that transforms an image, performed by a computer system, the method comprising:
generating first training data by performing data augmentation on at least one first image that includes a texture corresponding to a target style and generating second training data by performing data augmentation on at least one second image that is a target image for transformation to the target style and color transformation, the second training data including a transformation image for training in which the second image is color-transformed to an arbitrary color; and
training the transformation model to generate a texture-transformed image in which the second image is transformed to the texture corresponding to the target style and a color-transformed image in which the second image is transformed to a color corresponding to target color information, using the training data,
wherein a result image in which the second image is transformed according to the target style and the target color information is generated by synthesizing the texture-transformed image and the color-transformed image,
the transformation image is generated by transformation operations including an operation of transforming at least one of the second image and a color map of the second image generated by performing simplification processing on the second image to an image of a hue saturation value (HSV) color space, and an operation of color-transforming the image of the HSV color space to the arbitrary color, and
the trained transformation model is configured to transform an input image to represent the texture corresponding to the target style and the color corresponding to the input target color information without receiving an input of a reference image that represents the target style.