Realistic neural network based image style transfer

1. A method comprising:

receiving an indication of an instruction to modify an initial image comprising a user face, the instruction indicating to change the user face from having a first appearance to having a second appearance;

detecting the user face in the initial image;

separating the initial image into a cropped portion of the initial image and a non-cropped portion of the initial image, the cropped portion of the initial image comprising the user face;

modifying the cropped portion using a convolutional neural network, the modified cropped portion displaying the user face having the second appearance; and

modifying the non-cropped portion by applying adjustments to the non-cropped portion, wherein the adjustments are effects selected based on the convolutional neural network used to modified to the cropped portion.

2. The method of claim 1 wherein the convolutional neural network is trained on a set of images including images of user faces exhibiting the first appearance and images of user faces exhibiting the second appearance.

3. The method of claim 1 further comprising:

generating a result image by blending the modified cropped portion with the modified non-cropped portion.

4. The method of claim 1 further comprising:

generating a result image by blending the modified cropped portion with the modified non-cropped portion and the initial image.

5. The method of claim 4 wherein the result image is generated by blending modified cropped portion with the modified non-cropped portion and the initial image using Laplacian blending.

6. The method of claim 4 further comprising:

storing the result image.

7. The method of claim 1 , wherein the convolutional neural network comprises a plurality of downsampling convolution layers that input into a plurality of residual block layers.

8. The method of claim 7 wherein the method is performed on a client device, and wherein the method further comprising:

identifying a model type of the client device; and

decreasing a number of residual blocks implemented in the plurality of residual block layers based on the model type of the client device not satisfying a pre-specified computational resource threshold.

9. The method of claim 1 further comprising:

selecting the convolutional neural network from a plurality of convolution neural networks, wherein the convolution neural network was trained to change user faces to the second appearance.

10. The method of claim 1 wherein the second appearance is an aged face and the effects selected comprise an effect to color hair in the non-cropped portion to appear gray.

11. The method of claim 1 wherein the method is performed on a mobile device, and wherein the modified cropped portion is generated on the mobile device.

12. The method of claim 1 , further comprising:

adjusting color values of the modified non-cropped portion spatially close to the modified cropped portion.

13. The method of claim 12 wherein the adjusting color values comprises:

transferring the modified non-cropped portion from a first color space to a second color space;

transferring the modified cropped portion from the first color space to the second color space; and

adjusting the second color space of the modified non-cropped portion to colors that are closer to the second color space of the modified cropped portion.

14. The method of claim 13 , further comprising:

transferring the modified non-cropped portion from the second color space to the first color space.

15. The method of claim 14 , wherein the first color space is a red, green, and blue (RGB) color space and the second color space is a luma component (Y), two chrominance components (U) and red projection (V) (YUV) color space, and wherein adjustments are implemented using a histogram matching scheme that adjusts a distribution of the second color space of the non-cropped portion.

16. The method of claim 1 further comprising:

generating a result image by blending the modified cropped portion with the modified non-cropped portion; and

publishing the result image as an ephemeral message on a social network site.

17. A system comprising:

one or more processors of a client device;

an image sensor; and

a memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:

receiving an indication of an instruction to modify an initial image comprising a user face, the instruction indicating to change the user face from having a first appearance to having a second appearance;

detecting the user face in the initial image;

separating the initial image into a cropped portion of the initial image and a non-cropped portion of the initial image, the cropped portion of the initial image comprising the user face;

modifying the cropped portion using a convolutional neural network, the modified cropped portion displaying the user face having the second appearance; and

modifying the non-cropped portion by applying adjustments to the non-cropped portion, wherein the adjustments are effects selected based on the convolutional neural network used to modified to the cropped portion.

18. The system of claim 17 wherein the convolutional neural network is trained on a set of images including images of user faces exhibiting the first appearance and images of user faces exhibiting the second appearance.

19. A computer-readable storage medium embodying instructions that, when executed by a device, cause the device to perform operations comprising:

receiving an indication of an instruction to modify an initial image comprising a user face, the instruction indicating to change the user face from having a first appearance to having a second appearance;

detecting the user face in the initial image;

separating the initial image into a cropped portion of the initial image and a non-cropped portion of the initial image, the cropped portion of the initial image comprising the user face;

modifying the cropped portion using a convolutional neural network, the modified cropped portion displaying the user face having the second appearance; and

modifying the non-cropped portion by applying adjustments to the non-cropped portion, wherein the adjustments are effects selected based on the convolutional neural network used to modified to the cropped portion.

20. The computer-readable storage medium of claim 19 wherein the convolutional neural network is trained on a set of images including images of user faces exhibiting the first appearance and images of user faces exhibiting the second appearance.