Personalized machine learning system to edit images based on a provided style
Machine learning (ML) models are introduced for image stylization, which learn and apply multiple selectable image styles, including styles learned in an ad hoc fashion. According to some embodiments, such models may be trained on images or image pairs comprising images stylized into one or more of a plurality of predetermined styles. At inference time, a style vector representative of a particular selected style may be obtained and injected into the neural network at one or more locations to stylize an input image into the selected style. According to other embodiments, the neural network may be trained in an ad hoc fashion to learn new styles based on small sets of input images. Adversarial training (e.g., in the form of a discriminator network and/or conditional generative adversarial network (C-GAN) loss) may optionally be incorporated into the training to reduce artifacts and generate images that more closely match the selected style.
1. A non-transitory program storage device (NPSD) comprising computer readable instructions executable by one or more processors to:
obtain a neural network configured to stylize images into a plurality of styles, wherein the neural network is trained on images comprising images stylized into one of the plurality of styles;
obtain an input image;
obtain a selection of a first style of the plurality of styles;
obtain a style vector representative of the first style;
inject the style vector into the neural network at a first location; and
use the neural network with the injected style vector to stylize at least a portion of the input image into a stylized output image, based on the first style.
2. The NPSD of claim 1 , comprising further computer readable instructions executable by one or more processors to: inject the style vector into the neural network at one or more second locations.
3. The NPSD of claim 1 , comprising further computer readable instructions executable by the one or more processors to: identify one or more semantic regions within the input image, wherein the computer readable instructions to stylize at least a portion of the input image into a stylized output image based on the first style further comprise computer readable instructions to stylize only the identified one or more semantic regions within the input image.
4. The NPSD of claim 1 , wherein the style vector representative of the first style is obtained from an embedding matrix.
5. The NPSD of claim 1 , wherein the first location comprises one of: a bottleneck layer of the neural network; a downsampling layer of the neural network; or an upsampling layer of the neural network.
6. The NPSD of claim 1 , wherein the style vector is injected into the neural network in the form of an additional feature channel.
7. The NPSD of claim 6 , wherein the additional feature channel is concatenated with an existing feature map at the first location in the neural network.
8. A system, comprising:
one or more processors; and
one or more non-transitory program storage devices comprising computer readable instructions executable by the one or more processors to:
obtain a neural network configured to stylize images into a plurality of styles, wherein the neural network is trained on images comprising images stylized into one of the plurality of styles;
obtain an input image;
obtain a selection of a first style of the plurality of styles;
obtain a style vector representative of the first style;
inject the style vector into the neural network at a first location; and
use the neural network with the injected style vector to stylize at least a portion of the input image into a stylized output image, based on the first style.
9. The system of claim 8 , wherein the computer readable instructions further comprise computer readable instructions executable by one or more processors to: inject the style vector into the neural network at one or more second locations.
10. The system of claim 8 , comprising further computer readable instructions executable by the one or more processors to: identify one or more semantic regions within the input image, wherein the computer readable instructions to stylize at least a portion of the input image into a stylized output image based on the first style further comprise computer readable instructions to stylize only the identified one or more semantic regions within the input image.
11. The system of claim 8 , wherein the style vector representative of the first style is obtained from an embedding matrix.
12. The system of claim 8 , wherein the first location comprises one of: a bottleneck layer of the neural network; a downsampling layer of the neural network; or an upsampling layer of the neural network.
13. The system of claim 12 , wherein the style vector is injected into the neural network in the form of an additional feature channel.
14. A method, comprising:
obtaining a training set comprising a first plurality of images, wherein the first plurality of images comprises a second plurality of image pairs, wherein each image pair comprises a first image and a version of the first image stylized into one of a plurality of styles;
initializing a neural network configured to learn the plurality of styles from the first plurality of images, wherein the neural network comprises a style embedding matrix having an entry for each style of the plurality of styles; and
for each image pair in the second plurality of image pairs:
determining a style of the stylized version of the first image from the current image pair;
decomposing the first image from the current image pair into a first plurality of features using the neural network;
reconstructing the first image from the current image pair using a style vector representative of the determined style to attempt to match the stylized version of the first image from the current image pair; and
refining the neural network based, at least in part, on one or more loss functions computed between the reconstructed first image from the current image pair and the stylized version of the first image from the current image pair, wherein refining the neural network further comprises updating the style embedding matrix entry for the determined style of the first image from the current image pair.
15. The method of claim 14 , wherein at least one of the one or more loss functions computed between each reconstructed first image and the stylized version of each first image comprises an adversarial loss term for the reconstructed first image.
16. The method of claim 14 , further comprising:
for each image pair in the second plurality of image pairs:
performing an instance normalization operation on the first image from the current image pair.
17. The method of claim 16 , wherein the instance normalization operation comprises a conditional instance normalization (CIN) operation based, at least in part, on the determined style of the stylized version of the first image from the current image pair.
18. The method of claim 14 , further comprising:
obtaining a third plurality of images, wherein the third plurality of images comprise images stylized in an ad hoc style that is not among the plurality of styles; and
for each image in the third plurality of images:
performing a conditional instance normalization (CIN) operation on the current image from the third plurality of images; and
refining the neural network based, at least in part, on one or more parameters learned from the current image of the third plurality of images to learn the ad hoc style.
19. The method of claim 18 , wherein at least a portion of the parameters learned relating to the ad hoc style is transmitted via a network to a server for aggregation into a global style model.
20. The method of claim 14 , wherein each image pair further comprises a version of the first image stylized into each of the plurality of styles, and wherein:
for each image pair in the second plurality of image pairs, the method further comprises:
determining a style of each of the stylized versions of the first image from the current image pair;
decomposing the first image from the current image pair into a first plurality of features using the neural network;
reconstructing the first image from the current image pair a plurality of times, using a style vector representative of the corresponding determined style each time, to attempt to match each of the stylized versions of the first image from the current image pair; and
refining the neural network based, at least in part, on one or more loss functions computed between each of the reconstructed versions of the first image from the current image pair and the corresponding stylized version of the first image from the current image pair, wherein refining the neural network further comprises updating the corresponding style embedding matrix entry for the style of the respective stylized version of the first image from the current image pair.