Generating stylized images on mobile devices
Methods, systems, and non-transitory computer readable media are disclosed for generating artistic images by applying an artistic-effect to one or more frames of a video stream or digital images. In one or more embodiments, the disclosed system captures a video stream utilizing a camera of a computing device. The disclosed system deploys a distilled artistic-effect neural network on the computing device to generate an artistic version of the captured video stream at a first resolution in real time. The disclosed system can provide the artistic video stream for display via the computing device. Based on an indication of a capture event, the disclosed system utilizes the distilled artistic-effect neural network to generate an artistic image at a higher resolution than the artistic video stream. Furthermore, the disclosed system tunes and utilizes an artistic-effect patch generative adversarial neural network to modify parameters for the distilled artistic-effect neural network.
1. A system comprising:
a camera;
one or more memory devices comprising an artistic-effect neural network; and
at least one processing device coupled to the one or more memory devices, the at least one processing device to perform operations comprising:
capturing, utilizing the camera, a video stream comprising a plurality of frames;
generating, in real-time utilizing the artistic-effect neural network, a synthesized artistic video stream at a first resolution by applying an artistic-effect to the plurality of frames from the video stream;
providing, for display via a viewfinder as the video stream is being captured, the synthesized artistic video stream at the first resolution; and
generating, based on an indication of a capture event and utilizing the artistic-effect neural network, an artistic image comprising a frame from the synthesized artistic video stream with the artistic-effect at a second resolution that is higher than the first resolution.
2. The system as recited in claim 1 , wherein generating, based on the indication of the capture event and utilizing the artistic-effect neural network, the artistic image comprises generating the artistic image at a 4K resolution.
3. The system as recited in claim 1 , wherein the system comprises a mobile computing device.
4. The system as recited in claim 3 , wherein the artistic-effect comprises an anime style.
5. The system as recited in claim 3 , wherein the artistic-effect neural network comprises convolutional blocks comprising a single upsampling block and efficient separable convolutions.
6. The system as recited in claim 3 , wherein the operations further comprise generating, based on an indication of a selection of a second artistic-effect, in real time utilizing a second artistic-effect neural network, a second synthesized artistic video stream at the first resolution by applying the second artistic-effect to the plurality of frames from the video stream.
7. A non-transitory computer-readable medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
capturing, utilizing a camera, a video stream comprising a plurality of frames;
generating, utilizing an artistic-effect neural network, a synthesized artistic video stream at a first resolution by applying an artistic-effect to the plurality of frames from the video stream;
providing, for display via a viewfinder, the synthesized artistic video stream at the first resolution as the video stream is being captured; and
generating, based on an indication of a capture event and utilizing the artistic-effect neural network, an artistic image comprising a frame from the synthesized artistic video stream with the artistic-effect at a second resolution that is higher than the first resolution.
8. The non-transitory computer-readable medium of claim 7 , wherein generating, in real-time utilizing the artistic-effect neural network, the synthesized artistic video stream at the first resolution allows for a real-time display of the synthesized artistic video stream as the video stream is being captured.
9. The non-transitory computer-readable medium of claim 7 , wherein generating, utilizing the artistic-effect neural network, the artistic image at the second resolution comprises generating the artistic image at a resolution that is twice the first resolution.
10. The non-transitory computer-readable medium of claim 7 , wherein the operations further comprise generating and displaying via a graphical user interface including the viewfinder, a plurality of different artistic effect previews; and
wherein generating, utilizing the artistic-effect neural network, the synthesized artistic video stream at the first resolution by applying the artistic-effect to the plurality of frames from the video stream is in response to a selection of an artistic effect preview corresponding to an artistic effect.
11. The non-transitory computer-readable medium of claim 10 , wherein the operations further comprise:
receiving a selection of a second artistic effect preview;
generating, based on the selection of the second artistic effect preview, utilizing a second artistic-effect neural network, a second synthesized artistic video stream at the first resolution by applying a second artistic-effect to the plurality of frames from the video stream; and
replacing the display of the synthesized artistic video stream in the viewfinder with the second synthesized artistic video stream.
12. The non-transitory computer-readable medium of claim 7 , wherein the operations further comprise displaying the artistic image in place of the synthesized artistic video stream.
13. The non-transitory computer-readable medium of claim 7 , wherein generating, utilizing the artistic-effect neural network, the artistic image comprises upsampling within the artistic-effect neural network.
14. A method comprising:
capturing, utilizing a camera, a video stream comprising a plurality of frames;
generating, in real-time utilizing an artistic-effect neural network, a synthesized artistic video stream at a first resolution by applying an artistic-effect to the plurality of frames from the video stream;
providing, for display via a viewfinder as the video stream is being captured, the synthesized artistic video stream at the first resolution; and
generating, based on an indication of a capture event and utilizing the artistic-effect neural network, an artistic image comprising a frame from the synthesized artistic video stream with the artistic-effect at a second resolution that is higher than the first resolution.
15. The method of claim 14 , wherein generating, based on the indication of the capture event and utilizing the artistic-effect neural network, the artistic image comprises generating the artistic image at a 4K resolution.
16. The method of claim 14 , wherein:
capturing, utilizing the camera, the video stream comprises capturing the video stream utilizing a mobile phone; and
providing, for display via the viewfinder, the synthesized artistic video stream, comprises displaying the synthesized artistic video stream on the mobile phone.
17. The method of claim 14 , wherein generating, the artistic image with the artistic-effect comprises generating an artistic image with a painting style.
18. The method of claim 14 , further comprising generating, based on an indication of a selection of a second artistic-effect, in real time utilizing a second artistic-effect neural network, a second synthesized artistic video stream at the first resolution by applying the second artistic-effect to the plurality of frames from the video stream.
19. The method of claim 14 , further comprising generating and displaying via a graphical user interface including the viewfinder, a plurality of different artistic effect previews; and
wherein generating, in real-time utilizing the artistic-effect neural network, the synthesized artistic video stream at the first resolution by applying the artistic-effect to the plurality of frames from the video stream is in response to a selection of an artistic effect preview corresponding to an artistic effect.
20. The method of claim 14 , wherein generating, utilizing the artistic-effect neural network, the artistic image comprises upsampling within the artistic-effect neural network.