Multi-dimensional image stylization using transfer learning
A method for generating a multi-dimensional stylized image. The method includes providing input data into a latent space for a style conditioned multi-dimensional generator of a multi-dimensional generative model and generating the multi-dimensional stylized image from the input data by the style conditioned multi-dimensional generator. The method further includes synthesizing content for the multi-dimensional stylized image using a latent code and corresponding camera pose from the latent space to formulate an intermediate code to modulate synthesis convolution layers to generate feature images as multi-planar representations and synthesizing stylized feature images of the feature images for generating the multi-dimensional stylized image of the input data. The style conditioned multi-dimensional generator is tuned using a guided transfer learning process using a style prior generator.
1 . A method for generating a multi-dimensional stylized image, the method comprising:
providing input data into a latent space for a style conditioned multi-dimensional generator of a multi-dimensional generative model; and
generating the multi-dimensional stylized image from the input data by the style conditioned multi-dimensional generator by:
synthesizing content for the multi-dimensional stylized image using a latent code and corresponding camera pose from the latent space to formulate an intermediate code to modulate synthesis convolution layers to generate feature images as multi-planar representations; and
synthesizing stylized feature images of the feature images for generating the multi-dimensional stylized image of the input data,
wherein the style conditioned multi-dimensional generator is tuned using a guided transfer learning process using a style prior generator, wherein the style prior generator is a two-dimensional generator trained on style exemplars having a stylization and configured to generated stylized two-dimensional images of realistic images, and wherein the style prior generator is conditioned using a transfer learning process using a predetermined amount of style exemplars by synthesizing the style exemplars into feature maps having the stylization, wherein the predetermined amount of style exemplars is 10 or more.
2 . The method of claim 1 , wherein the guided transfer learning process includes:
generating two-dimensional stylized images of training data using the style prior generator;
generating the multi-dimensional stylized images of the training data using the style conditioned multi-dimensional generator; and
discriminating the two-dimensional stylized images and the multi-dimensional stylized images using a discriminator that compares a distribution of translated images of the multi-dimensional stylized images to a distribution of the two-dimensional stylized images to tune the style conditioned multi-dimensional generator to generate the stylized feature images.
3 . The method of claim 1 , wherein the feature images are further sampled into a neural radiance field at predetermined camera poses and accumulated to synthesize generation of raw feature images using volume rendering.
4 . The method of claim 3 , further comprising upsampling the volume rendered feature images, wherein the upsampling includes refining the volume rendered feature images using a super resolution module to synthesize RGB images of the multi-dimensional stylized image.
5 . The method of claim 1 , wherein the latent code and corresponding camera pose are passed through a mapping network to formulate the intermediate code.
6 . The method of claim 1 , wherein the predetermined amount of style exemplars is between 10 and 20.
7 . The method of claim 1 , wherein during the transfer learning process realistic images are used, wherein camera poses are determined from the realistic images.
8 . The method of claim 1 , wherein the stylization is an artistic style selected from the group consisting of cartoon, oil painting, cubism, abstract, comic, Sam Yang, sculpture, and anime.
9 . The method of claim 1 , wherein the providing the input data includes inverting the input data using an encoder that outputs latent codes, wherein the style conditioned multi-dimensional generator is configured to generate the multi-planar representations using the encoded intermediate latent code.
10 . The method of claim 9 , wherein the encoder is trained by:
randomly sampling a plurality of latent codes from a Gaussian-distributed latent space with a fixed front camera pose,
feeding the plurality of latent codes and the fixed front camera pose into a mapping network to obtain intermediate latent code samples,
training the encoder with the obtained intermediate latent code samples to encode intermediate latent codes.
11 . The method of claim 1 , wherein the multi-dimensional stylized image is a three-dimensional stylized portrait image and the input data is a self-portrait image.
12 . A computing system comprising:
one or more processors; and
one or more computer-readable media having computer-executable instructions stored thereon that, upon execution, cause the one or more processors to implement a multi-dimensional generative adversarial network (GAN), the GAN comprising:
an encoder for inverting input data into a latent space;
a style conditioned multi-dimensional generator; and
a style prior generator,
wherein the style conditioned multi-dimensional generator is configured to:
synthesize content for a multi-dimensional stylized image using a latent code and corresponding camera pose from the latent space to formulate an intermediate code to modulate synthesis convolution layers to generate feature images as multi-planar representations;
synthesize stylized feature images of the feature images as the multi-planar representations to generate the multi-dimensional stylized image of the input data,
wherein the style conditioned multi-dimensional generator is tuned using a guided transfer learning process using the style prior generator, wherein the style prior generator is a two-dimensional generator trained on style exemplars having a stylization and configured to generated stylized two-dimensional images of realistic images, and wherein the style prior generator is conditioned using a transfer learning process using a predetermined amount of style exemplars by synthesizing the style exemplars into feature maps having the stylization, wherein the predetermined amount of style exemplars is 10 or more.
13 . The computing system of claim 12 , further comprising a decoder, the decoder including a super resolution module to synthesize RGB images of the multi-dimensional stylized image.
14 . The computing system of claim 12 , further comprising a discriminator, wherein the style prior generator is configured to generate two-dimensional stylized images of training data and the discriminator is configured to discriminate between a distribution of the two-dimensional stylized images from the style prior generator and a distribution of translated images of multi-dimensional stylized images of the training data from the style conditioned multi-dimensional generator to tune the style conditioned multi-dimensional generator.
15 . The computing system of claim 14 , wherein the style prior generator is a two-dimensional generator trained on style exemplars having a stylization and configured to generate stylized two-dimensional images of realistic images.
16 . The computing system of claim 15 , wherein the stylization is an artistic style selected from the group consisting of cartoon, oil painting, cubism, abstract, comic, Sam Yang, sculpture, and anime.
17 . The computing system of claim 12 , wherein the multi-dimensional stylized image is a three-dimensional stylized portrait image and the input data is a self-portrait image.
18 . A non-transitory computer-readable medium having computer-executable instructions stored thereon that, upon execution, cause one or more processors to perform operations comprising:
providing input data into a latent space for a style conditioned multi-dimensional generator of a multi-dimensional generative model; and
generating a multi-dimensional stylized image from the input data by the style conditioned multi-dimensional generator by:
synthesizing content for the multi-dimensional stylized image using a latent code and corresponding camera pose from the latent space to formulate an intermediate code to modulate synthesis convolution layers to generate feature images as multi-planar representations;
synthesizing stylized feature images of the feature images for generating the multi-dimensional stylized image of the input data,
wherein the style conditioned multi-dimensional generator is tuned using a guided transfer learning process using a style prior generator, wherein the style prior generator is a two-dimensional generator trained on style exemplars having a stylization and configured to generated stylized two-dimensional images of realistic images, and wherein the style prior generator is conditioned using a transfer learning process using a predetermined amount of style exemplars by synthesizing the style exemplars into feature maps having the stylization, wherein the predetermined amount of style exemplars is 10 or more.
19 . The non-transitory computer-readable medium according to claim 18 , wherein the operations further include:
generating two-dimensional stylized images of training data using the style prior generator;
generating the multi-dimensional stylized images of the training data using the style conditioned multi-dimensional generator; and
discriminating the two-dimensional stylized images and the multi-dimensional stylized images using a discriminator that compares a distribution of translated images of the multi-dimensional stylized images to a distribution of the two-dimensional stylized images to tune the style conditioned multi-dimensional generator to generate the stylized feature images.