Deep feature generative adversarial neural networks
A mobile device can implement a neural network-based domain transfer scheme to modify an image in a first domain appearance to a second domain appearance. The domain transfer scheme can be configured to detect an object in the image, apply an effect to the image, and blend the image using color space adjustments and blending schemes to generate a realistic result image. The domain transfer scheme can further be configured to efficiently execute on the constrained device by removing operational layers based on resources available on the mobile device.
1. A method comprising:
identifying an image on a user device;
identifying an instruction to apply an image effect to the image;
generating, by a processor of the user device, a modified image by applying a domain transfer neural network to the image, the domain transfer neural network comprising an adversarial transformation network within a generator network;
storing the modified image; and
publishing, by the user device, the modified image to a network site as an ephemeral message.
2. The method of claim 1 , wherein the generator network comprises an encoder network and decoder network, the adversarial transformation network being between the encoder network and the decoder network.
3. The method of claim 2 , wherein the adversarial transformation network is adversarially trained using a discrimination network that receives output data from the decoder network.
4. The method of claim 2 , wherein the domain transfer neural network is trained in stages comprising:
a first stage in which at least one of the encoder network or the decoder network are trained; and
a second stage in which the adversarial transformation network is using a discrimination network.
5. The method of claim 4 , wherein in the first stage the encoder network and the decoder network are jointly trained using feature loss.
6. The method of claim 4 , wherein the adversarial transformation network is not trained in the first stage.
7. The method of claim 4 , wherein the encoder network and the decoder network are not trained in the second stage.
8. The method of claim 4 , wherein the adversarial transformation network is trained using Generative Adversarial Loss (GAN) loss.
9. The method of claim 1 , wherein the adversarial transformation network is trained to transfer an image between different domains.
10. The method of claim 9 , wherein the domains includes one or more of the following: an old person domain, a young person domain, a masculine person domain, a feminine person domain, a painted domain, a photorealistic domain.
11. The method of claim 1 , wherein the domain transfer neural network is a convolutional neural network.
12. The method of claim 1 , further comprising:
receiving, from a server, the adversarial transformation network pre-trained to apply the image effect.
13. The method of claim 1 , further comprising:
receiving, from an input interface of the user device, the instruction to apply the image effect on the image.
14. The method of claim 13 , further comprising:
in response to receiving the instruction, selecting, from a plurality of domain transfer neural networks stored on the user device, the domain transfer neural network based on the domain transfer neural network being trained for the image effect specified by the instruction, wherein each of the plurality of domain transfer neural networks is trained for different image effects using adversarial transformation networks within respective generator networks.
15. The method of claim 1 , further comprising:
generating the image using an image sensor of the user device.
16. A system comprising:
one or more processors of a client device;
an image sensor; and
a memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
identifying an image on a user device;
identifying an instruction to apply an image effect to the image;
generating a modified image by applying a domain transfer neural network to the image, the domain transfer neural network comprising an adversarial transformation network within a generator network;
storing the modified image; and
publishing, by the user device, the modified image to a network site as an ephemeral message.
17. The system of claim 16 , wherein the generator network comprises an encoder network and decoder network, the adversarial transformation network being between the encoder network and the decoder network.
18. A machine-readable storage device embodying instructions that, when executed by a device, cause the device to perform operations comprising:
identifying an image on a user device;
identifying an instruction to apply an image effect to the image;
generating a modified image by applying a domain transfer neural network to the image, the domain transfer neural network comprising an adversarial transformation network within a generator network;
storing the modified image; and
publishing, by the user device, the modified image to a network site as an ephemeral message.