Enhancing detailed segments in latent code-based edited digital images
Methods, systems, and non-transitory computer readable media are disclosed for intelligently enhancing details in edited images. The disclosed system iteratively updates residual detail latent code for segments in edited images where detail has been lost through the editing process. More particularly, the disclosed system enhances an edited segment in an edited image based on details in a detailed segment of an image. Additionally, the disclosed system may utilize a detail neural network encoder to project the detailed segment and a corresponding segment of the edited image into a residual detail latent code. In some embodiments, the disclosed system generates a refined edited image based on the residual detail latent code and a latent vector of the edited image.
1. A system comprising:
a storage device storing an image and an edited image generated from the image;
one or more memory devices comprising a detail neural network encoder and a neural network generator; and
at least one processor configured to cause the system to:
extract a detailed segment from the image;
extract an edited segment from the edited image, wherein the edited segment comprises an edited version of the detailed segment;
generate a residual detail latent code by projecting the detailed segment and the edited segment into a latent space utilizing the detail neural network encoder; and
generate a refined edited image that depicts the edited image comprising a refined edited segment with an increased level of detail by synthesizing the residual detail latent code and a latent vector of the edited image utilizing the neural network generator.
2. The system as recited in claim 1 , wherein the at least one processor is configured to cause the system to extract the detailed segment from the image and the edited segment from the edited image utilizing a segmentation neural network.
3. The system as recited in claim 1 , wherein the at least one processor is configured to cause the system to iteratively refine the edited image utilizing updated residual detail latent codes generated from previous iterations of the refined edited segment.
4. The system as recited in claim 3 , wherein the at least one processor is configured to cause the system to iteratively refine the refined edited segment of the refined edited image by:
combining the refined edited segment and the detailed segment of the image;
projecting the combined refined edited segment and the detailed segment into an updated residual detail latent code utilizing the detail neural network encoder; and
updating the refined edited image by synthesizing the updated residual detail latent code and the latent vector of the refined edited image utilizing the neural network generator.
5. The system as recited in claim 1 , wherein the at least one processor is configured to cause the system to extract the detailed segment from the image by segmenting a portion of the image comprising at least one of a hair or facial wrinkles.
6. The system as recited in claim 1 , wherein the at least one processor is configured to cause the system to:
generate a total loss based on the refined edited image, the edited image, and the image; and
modify parameters of the detail neural network encoder based on the total loss.
7. The system as recited in claim 6 , wherein the at least one processor is configured to cause the system to generate the total loss by:
generating a global consistency loss to constrain differences between a non-detailed segment from the edited image and a non-detailed segment from the refined edited image; and
generating a local similarity loss to constrain differences between the detailed segment from the image and the refined edited segment from the refined edited image.
8. The system as recited in claim 7 , wherein the at least one processor is configured to cause the system to generate the local similarity loss by:
sampling a patch at a location in the detailed segment from the image;
sampling a refined patch at a corresponding location of the refined edited segment from the refined edited image; and
comparing the patch and the refined patch.
9. The system as recited in claim 8 , wherein sampling the refined patch at the corresponding location comprises sampling the refined patch from a neighborhood area of the location in the refined edited segment.
10. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
combining a detailed segment from an image with an edited segment from an edited image, wherein the edited image is generated from the image, and wherein the edited segment comprises an edited version of the detailed segment;
projecting the combined detailed segment and edited segment into a residual detail latent code utilizing a detail neural network encoder;
generating a refined edited image that depicts the edited image comprising a refined edited segment with an increased level of detail by synthesizing the residual detail latent code and a latent vector of the edited image utilizing a neural network generator; and
iteratively refining the refined edited segment of the refined edited image.
11. The non-transitory computer readable medium recited in claim 10 , wherein iteratively refining the refined edited segment of the refined edited image comprises:
combining the refined edited segment and the detailed segment of the image;
projecting the combination of the refined edited segment and the detailed segment of the image into an updated residual detail latent code utilizing the detail neural network encoder; and
updating the refined edited image by synthesizing the updated residual detail latent code and a latent vector of the edited image utilizing the neural network generator.
12. The non-transitory computer readable medium recited in claim 10 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising extracting the detailed segment from the image by segmenting a portion of the image comprising hair.
13. The non-transitory computer readable medium recited in claim 10 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
generating a total loss based on the refined edited image, the edited image, and the image; and
modifying parameters of the detail neural network encoder based on the total loss.
14. The non-transitory computer readable medium recited in claim 13 , wherein generating the total loss comprises:
generating a global consistency loss to constrain differences between a non-detailed segment from the edited image and a non-detailed segment from the refined edited image; and
generating a local similarity loss to constrain differences between the detailed segment from the image and the refined edited segment from the refined edited image.
15. The non-transitory computer readable medium recited in claim 13 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to fix the parameters of the neural network generator when modifying parameters of the detail neural network encoder based on the total loss.
16. The non-transitory computer readable medium recited in claim 10 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising extracting the detailed segment from the image and the edited segment from the edited image by utilizing a segmentation model.
17. The non-transitory computer readable medium recited in claim 10 , further comprising instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising generating the edited image from the image utilizing latent-code based editing.
18. A method of generating a refined edited image comprising:
extracting a detailed segment from an image and an edited detailed segment from an edited image, wherein the edited image is generated from the image, and wherein the edited detailed segment comprises an edited version of the detailed segment;
performing a step for refining the edited image based on the detailed segment and the edited detailed segment; and
providing the refined edited image for display.
19. The method recited in claim 18 , further comprising:
projecting the image into a latent space to generate a latent vector for the image;
modifying the latent vector of the image; and
generating the edited image by reconstructing the modified latent vector utilizing a neural network generator.
20. The method recited in claim 18 , wherein the step for refining the edited image comprises iteratively refining the edited image.