Hierarchical scale matching and patch estimation for image style transfer with arbitrary resolution
A style of a digital image is transferred to another digital image of arbitrary resolution. A high-resolution (HR) content image is segmented into several low-resolution (LR) patches. The resolution of a style image is matched to have the same resolution as the LR content image patches. Style transfer is then performed on a patch-by-patch basis using, for example, a pair of feature transforms—whitening and coloring. The patch-by-patch style transfer process is then repeated at several increasing resolutions, or scale levels, of both the content and style images. The results of the style transfer at each scale level are incorporated into successive scale levels up to and including the original HR scale. As a result, style transfer can be performed with images having arbitrary resolutions to produce visually pleasing results with good spatial consistency.
1. In a digital medium environment for editing digital images, a computer-implemented method of transferring a style from one digital image to another digital image, the method comprising:
segmenting, by at least one processor of a computing device, an original digital content image at a first content scale into a first plurality of patches at the first content scale, wherein the first content scale is lower than an original scale of the original digital content image;
transferring, by the at least one processor, a style of a digital style image at a first style scale to each of the patches of the original digital content image at the first content scale based on a covariance matrix of a content feature map of each of the first plurality of patches, the content feature map being a function of a mean of features of each of the first plurality of patches, thereby providing a first modified digital content image at the first content scale;
upscaling, by the at least one processor, the first modified digital content image from the first content scale to a second content scale, the second content scale being greater than the first content scale;
combining, by the at least one processor, the original digital content image at the second content scale with the first modified digital content image at the second content scale;
segmenting, by the at least one processor, the combined digital content image at the second content scale into a second plurality of patches at the second content scale;
transferring, by the at least one processor, the style of the digital style image at a second style scale to each of the patches of the combined digital content image at the second content scale, the second style scale being greater than the first style scale, thereby providing a second modified digital content image at the second content scale; and
causing, by the at least one processor, display of an output image with content of the second modified digital content image at the second content scale in the style of the digital style image.
2. The method of claim 1 , wherein transferring the style of the digital style image at either or both of the first and second content scales to each of the first and second plurality of patches, respectively, includes performing whitening and coloring transforms on the respective ones of the first and second plurality of patches.
3. The method of claim 2 , wherein the whitening transform includes applying a first linear transformation to the content feature map, wherein the coloring transform includes applying a second linear transformation to the results of the whitening transform, wherein the first linear transformation is determined from a decomposition of the covariance matrix of the content feature map, and wherein the second linear transformation is determined from a decomposition of a covariance matrix of a style feature map.
4. The method of claim 3 , wherein the mean of the features in the content feature map is subtracted from features in each of the first and second plurality of patches, respectively.
5. The method of claim 1 , wherein combining the digital content image at the second content scale with the modified digital content image at the second content scale includes performing a linear combination of the digital content image at the second content scale with the modified digital content image at the second content scale.
6. The method of claim 1 , further comprising repeating the upscaling, combining, segmenting, and transferring at a third content scale and a third style scale, wherein the third content scale and the third style scale are each the same as the original scale of the original digital content image.
7. The method of claim 1 , wherein the second content scale is greater than the first content scale by a factor of two.
8. A non-transitory computer readable medium having instructions encoded thereon that when executed by at least one processor of a computing device cause the at least one processor to perform a process of transferring a style from one digital image to another digital image, the process comprising:
segmenting, by the at least one processor, an original digital content image at a first content scale into a first plurality of patches at the first content scale, wherein the first content scale is lower than an original scale of the original digital content image;
transferring, by the at least one processor, a style of a digital style image at a first style scale to each of the patches of the original digital content image at the first content scale based on a covariance matrix of a content feature map of each of the first plurality of patches, the content feature map being a function of a mean of features of each of the first plurality of patches, thereby providing a first modified digital content image at the first content scale;
upscaling, by the at least one processor, the first modified digital content image from the first content scale to a second content scale, the second content scale being greater than the first content scale;
combining, by the at least one processor, the original digital content image at the second content scale with the first modified digital content image at the second content scale;
segmenting, by the at least one processor, the combined digital content image at the second content scale into a second plurality of patches at the second content scale;
transferring, by the at least one processor, the style of the digital style image at a second style scale to each of the patches of the combined digital content image at the second content scale, the second style scale being greater than the first style scale, thereby providing a second modified digital content image at the second content scale; and
causing, by the at least one processor, display of an output image with content of the second modified digital content image at the second content scale in the style of the digital style image.
9. The non-transitory computer readable medium of claim 8 , wherein transferring the style of the digital style image at either or both of the first and second content scales to each of the first and second plurality of patches, respectively, includes performing whitening and coloring transforms on the respective ones of the first and second plurality of patches.
10. The non-transitory computer readable medium of claim 9 , wherein the whitening transform includes applying a first linear transformation to a content feature map of each of the respective first and second plurality of patches, wherein the coloring transform includes applying a second linear transformation to the results of the whitening transform, wherein the first linear transformation is determined from a decomposition of a covariance matrix of the content feature map, and wherein the second linear transformation is determined from a decomposition of a covariance matrix of a style feature map.
11. The non-transitory computer readable medium of claim 10 , wherein the mean of the features in the content feature map is subtracted from features in each of the first and second plurality of patches.
12. The non-transitory computer readable medium of claim 8 , wherein combining the digital content image at the second content scale with the modified digital content image at the second content scale includes performing a linear combination of the digital content image at the second content scale with the modified digital content image at the second content scale.
13. The non-transitory computer readable medium of claim 8 , wherein the process further comprises repeating the upscaling, combining, segmenting, and transferring at a third content scale and a third style scale, wherein the third content scale and the third style scale are each the same as the original scale of the original digital content image.
14. A system for transferring a style from one digital image to another digital image, the system comprising:
a storage; and
a processor operatively coupled to the storage, the processor configured to execute instructions stored in the storage that when executed cause the processor to carry out a process including
segmenting an original digital content image at a first content scale into a first plurality of patches at the first content scale, wherein the first content scale is lower than an original scale of the original digital content image;
transferring a style of a digital style image at a first style scale to each of the patches of the original digital content image at the first content scale based on a covariance matrix of a content feature map of each of the first plurality of patches, the content feature map being a function of a mean of features of each of the first plurality of patches, thereby providing a first modified digital content image at the first content scale;
upscaling the first modified digital content image from the first content scale to a second content scale, the second content scale being greater than the first content scale;
combining the original digital content image at the second content scale with the first modified digital content image at the second content scale;
segmenting the combined digital content image at the second content scale into a second plurality of patches at the second content scale;
transferring the style of the digital style image at a second style scale to each of the patches of the combined digital content image at the second content scale, the second style scale being greater than the first style scale, thereby providing a second modified digital content image at the second content scale; and
causing display of an output image with content of the second modified digital content image at the second content scale in the style of the digital style image.
15. The system of claim 14 , wherein transferring the style of the digital style image at either or both of the first and second content scales to each of the first and second plurality of patches, respectively, includes performing whitening and coloring transforms on the respective ones of the first and second plurality of patches.
16. The system of claim 15 , wherein the whitening transform includes applying a first linear transformation to a content feature map of each of the respective first and second plurality of patches, wherein the coloring transform includes applying a second linear transformation to the results of the whitening transform, wherein the first linear transformation is determined from a decomposition of a covariance matrix of the content feature map, and wherein the second linear transformation is determined from a decomposition of a covariance matrix of a style feature map.
17. The system of claim 16 , wherein the mean of the features in the content feature map is subtracted from features in each of the first and second plurality of patches, respectively.
18. The system of claim 14 , wherein combining the digital content image at the second content scale with the modified digital content image at the second content scale includes performing a linear combination of the digital content image at the second content scale with the modified digital content image at the second content scale.
19. The system of claim 14 , wherein the process further includes repeating the upscaling, combining, segmenting, and transferring at a third content scale and a third style scale, wherein the third content scale and the third style scale are each the same as the original scale of the original digital content image.
20. The system of claim 14 , wherein the second content scale is greater than the first content scale by a factor of two.