Plausible dayscale timelapse generation method and computing device
The disclosure relates to a field of plausible timelapse image(s) generation from a single image. A method of generating one or more images of a plausible dayscale timelapse sequence based on a content image using a trained generative neural network and a trained merging neural network is provided. The method includes receiving the content image and one of one or more predefined styles respectively corresponding to times of day to be applied to the content image or style images having styles to be applied to the content image, slicing the content image into n image crops, applying the trained generative neural network with each style to n image crops to obtain n image crops re-stylized according to each style, and merging the re-stylized n image crops for each style with the trained merging neural network to obtain images of a plausible dayscale timelapse sequence for the content image.
1 . A method performed by a computing device of generating one or more images of a plausible dayscale timelapse sequence based on a content image using a trained generative neural network and a trained merging neural network, the method comprising:
receiving, by the computing device, the content image and one of a predefined one or more styles to be applied to the content image or one or more style images having one or more styles to be applied to the content image;
slicing, by the computing device, the content image into n image crops by slicing each image of a set of training images into n strongly overlapping image crops in a predefined manner defined by an offset direction and a stride of k pixels;
applying, by the computing device, the trained generative neural network with each of one or more styles to n image crops to obtain n image crops re-stylized according to each of one or more styles; and
merging, by the computing device, the re-stylized n image crops for each of the one or more styles with the trained merging neural network to obtain one or more images of a plausible dayscale timelapse sequence for the content image,
wherein the re-stylized n image crops are generated without directly feeding a discriminator to the generative neural network,
wherein the trained generative neural network is initially and iteratively trained in a swap mode using the following operations repeatedly performed at a number of iterations:
inputting a first image as a training example into a content encoder to obtain a content tensor being a spatially smaller representation of content in the first image;
inputting a second image as a training example into a style encoder to obtain a style vector indicative of a style in the second image;
inputting the content tensor, the style vector into a decoder to obtain, using adaptive instance normalization, a third image having content of the first image and a style of the second image, and a segmentation mask corresponding to the first image;
alternately inputting the second image and the style vector or the style vector and the third image into a conditional discriminator configured to determine when an image is plausible enough under a given style; and
alternately updating parameters of the trained generative neural network and the conditional discriminator at different iterations based on a result of a determination by the conditional discriminator to ensure training of the trained generative neural network and conditional discriminator in an adversarial manner,
wherein in updating parameters of the trained generative neural network the segmentation mask is further considered, and
wherein the first image and the second image have a first resolution.
2 . The method of claim 1 , wherein the trained generative neural network is initially and iteratively trained in one or in a combination of the following modes: a swap mode, a random mode, or an autoencoder mode.
3 . The method of claim 2 ,
wherein the trained generative neural network is initially and iteratively trained in the random mode using the following operations repeatedly performed at a number of iterations:
inputting a first image as a training example into the content encoder to obtain a content tensor being a spatially smaller representation of content in the first image;
generating a random style vector from prior distribution;
inputting the content tensor, the random style vector into the decoder to obtain, using adaptive instance normalization, a third image having a content of the first image and a style defined by the random style vector, and a segmentation mask corresponding to the first image;
inputting the random style vector and the third image into a conditional discriminator configured to determine when the third image is plausible enough under a given style; and
updating parameters of the trained generative neural network based on a result of a determination by the conditional discriminator to ensure training of the trained generative neural network and conditional discriminator in an adversarial manner, and
wherein in updating parameters of the trained generative neural network the segmentation mask is further considered.
4 . The method of claim 2 ,
wherein the trained generative neural network is initially and iteratively trained in the autoencoder mode using the following operations repeatedly performed at a number of iterations:
inputting a first image as a training example into the content encoder to obtain a content tensor being a spatially smaller representation of content in the first image;
inputting the first image as a training example into the style encoder to obtain a style vector indicative of a style in the first image;
inputting the content tensor, the style vector into the decoder to obtain, using adaptive instance normalization, a third image having the content and a style of the first image, and a segmentation mask corresponding to the first image;
comparing the first image and the third image; and
updating parameters of the trained generative neural network based on a result of the comparison, and
wherein in updating parameters of the trained generative neural network the segmentation mask is further considered.
5 . The method of claim 4 , wherein the trained generative neural network comprises one or more dense skip connections configured to convey high frequency features not conveyed by the content tensor of the first image to the decoder.
6 . The method of claim 1 ,
wherein the merging neural network is initially and iteratively trained using the following operations repeatedly performed at a number of iterations:
obtaining a set of training images each having a second resolution
downsampling each image crop of the n image crops to a first resolution;
applying the trained generative neural network in an autoencoder mode to each image crop of the n image crops to obtain n translated crops each capturing artifacts and discrepancies between a corresponding image crop of an original training image and the translated image crop produced by the trained generative neural network;
inputting the n translated crops into the merging neural network to obtain a merged image with reduced artifacts and discrepancies, comparing the merged image with a corresponding original image of the set of training images; and
updating parameters of the merging neural network based on a result of the comparison, and
wherein the merged image has the second resolution.
7 . The method of claim 1 , wherein one or more styles respectively correspond to one or more times of day to be applied to the content image.
8 . A computing device comprising:
memory storing one or more computer programs; and
one or more processors communicatively coupled to the memory,
wherein the one or more computer programs include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the computing device to:
receive a content image and one of a predefined one or more styles to be applied to the content image or one or more style images having one or more styles to be applied to the content image,
slice the content image into n image crops by slicing each image of a set of training images into n strongly overlapping image crops in a predefined manner defined by an offset direction and a stride of k pixels,
apply a trained generative neural network with each of one or more styles to n image crops to obtain n image crops re-stylized according to each of the one or more styles, and
merge the re-stylized n image crops for each of one or more styles with the trained merging neural network to obtain one or more images of a plausible dayscale timelapse sequence for the content image, and
wherein the re-stylized n image crops are generated without directly feeding a discriminator to the generative neural network,
wherein the trained generative neural network is initially and iteratively trained in a swap mode using the following operations repeatedly performed at a number of iterations:
inputting a first image as a training example into a content encoder to obtain a content tensor being a spatially smaller representation of content in the first image;
inputting a second image as a training example into a style encoder to obtain a style vector indicative of a style in the second image;
inputting the content tensor, the style vector into a decoder to obtain, using adaptive instance normalization, a third image having content of the first image and a style of the second image, and a segmentation mask corresponding to the first image;
alternately inputting the second image and the style vector or the style vector and the third image into a conditional discriminator configured to determine when an image is plausible enough under a given style; and
alternately updating parameters of the trained generative neural network and the conditional discriminator at different iterations based on a result of a determination by the conditional discriminator to ensure training of the trained generative neural network and conditional discriminator in an adversarial manner,
wherein in updating parameters of the trained generative neural network the segmentation mask is further considered, and
wherein the first image and the second image have a first resolution.
9 . A method performed by a computing device of generating one or more images of a plausible dayscale timelapse sequence based on a content image using a trained generative neural network, the method comprising:
receiving, by the computing device, the content image and at least one predefined one or more styles to be applied to the content image or one or more style images having one or more styles to be applied to the content image;
reducing, by the computing device, a resolution of the content image to a lower resolution while maintaining an aspect ratio of the content image;
applying, by the computing device, the trained generative neural network with each of one or more styles to the reduced resolution of the content image to obtain one or more reduced content images re-stylized according to each of one or more styles;
decomposing, by the computing device, each of a re-stylized content image into high-frequency components and a low-frequency component having the lower resolution while maintaining the aspect ratio;
filtering, by the computing device, the low-frequency component by considering the content of a corresponding re-stylized content image; and
generating, by the computing device, one or more images of the plausible dayscale timelapse sequence based on the filtered low-frequency component and the high-frequency components of each of corresponding re-stylized content images,
wherein the obtained one or more reduced content images re-stylized according to each of one or more styles is obtained without directly feeding a discriminator to the generative neural network, and
wherein the generated one or more images of the plausible dayscale timelapse sequence are generated by applying a differentiated guided filter to the content image and a corresponding one of the obtained one or more reduced content images re-stylized according to each of one or more styles.
10 . The method of claim 9 , wherein a Laplacian pyramid is used for the decomposition.
11 . The method of claim 9 ,
wherein a guided filter is used for the filtering, and
wherein the content of the corresponding re-stylized content image is used as a guide for the filtering.
12 . The method of claim 9 , wherein the trained generative neural network is initially and repeatedly trained in one or in a combination of the following modes: a swap mode, a random mode, or an autoencoder mode.
13 . The method of claim 9 , wherein the trained generative neural network comprises a content encoder, a style encoder, and a decoder.
14 . The method of claim 13 , wherein the content encoder comprises two downsamplings and four residual blocks after each downsampling.
15 . The method of claim 13 , wherein the style encoder contains four downsampling blocks, and a downsampled result is averaged with respect to spatial information into a three-dimensional vector.
16 . The method of claim 13 , wherein the decoder comprises five residual blocks with an adaptive instance normalizations (AdaIN) therein and two upsampling blocks.