Utilizing a generative machine learning model to create modified digital images from an infill semantic map
The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For example, in one or more embodiments the disclosed systems utilize generative machine learning models to create modified digital images portraying human subjects. In particular, the disclosed systems generate modified digital images by performing infill modifications to complete a digital image or human inpainting for portions of a digital image that portrays a human. Moreover, in some embodiments, the disclosed systems perform reposing of subjects portrayed within a digital image to generate modified digital images. In addition, the disclosed systems in some embodiments perform facial expression transfer and facial expression animations to generate modified digital images or animations.
1. A non-transitory computer-readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
generating a semantic map corresponding to a digital image, wherein the semantic map comprises semantic classifications of pixels within the digital image;
generating, utilizing a semantic map diffusion neural network, an infill semantic map from the semantic map, wherein the infill semantic map comprises semantic classifications for an infill modification that indicates a region to fill for the digital image;
generating a diffusion representation of the digital image by utilizing diffusion layers of a digital image diffusion neural network; and
generating, utilizing denoising layers of the digital image diffusion neural network, a modified digital image from the diffusion representation and the infill semantic map.
2. The non-transitory computer-readable medium of claim 1 , wherein generating the modified digital image further comprises generating the modified digital image from the digital image by utilizing the digital image diffusion neural network conditioned on the infill semantic map.
3. The non-transitory computer-readable medium of claim 1 , wherein generating the infill semantic map comprises:
generating a diffusion representation of the semantic map utilizing diffusion layers of the semantic map diffusion neural network; and
generating, utilizing denoising layers of the semantic map diffusion neural network, the infill semantic map from the diffusion representation of the semantic map.
4. The non-transitory computer-readable medium of claim 3 , wherein the operations further comprise:
receiving a semantic editing input corresponding to the semantic map; and
generating the infill semantic map from the semantic map by conditioning the denoising layers of the semantic map diffusion neural network with the semantic editing input.
5. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise:
determine an input texture for modifying the digital image; and
generating the modified digital image from the infill semantic map by conditioning the denoising layers of the digital image diffusion neural network with the input texture.
6. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise:
generating semantic classifications for the infill semantic map; and
generating the modified digital image by conditioning the denoising layers of the digital image diffusion neural network with the semantic classifications.
7. A system comprising:
one or more memory devices comprising an input digital image, a semantic map model, a generative semantic machine learning model, and a generative image machine learning model; and
one or more processors configured to cause the system to:
generate, utilizing the semantic map model, a semantic map from the input digital image, wherein the semantic map comprises semantic classifications of pixels within the input digital image;
determine an infill modification indicating a region to fill for the input digital image;
generate, utilizing the generative semantic machine learning model, an infill semantic map from the infill modification and the semantic map, wherein the infill semantic map comprises semantic classifications for the infill modification that indicates the region to fill for the input digital image; and
generate, utilizing the generative image machine learning model, a modified digital image from the infill semantic map and the input digital image.
8. The system of claim 7 , wherein the one or more processors are configured to cause the system to:
determine an input texture for the region; and
generate the modified digital image from the input digital image by utilizing a digital image diffusion neural network conditioned on the infill semantic map and the input texture.
9. The system of claim 7 , wherein the one or more processors are configured to cause the system to:
segment, utilizing a segmentation neural network, an object to remove from the input digital image; and
generate the infill semantic map from the input digital image and the object to remove from the input digital image.
10. The system of claim 9 , wherein the one or more processors are configured to cause the system to generate the infill semantic map by generating, utilizing the generative semantic machine learning model, semantic classifications of pixels within the region to be filled.
11. The system of claim 8 , wherein the one or more processors are configured to cause the system to:
generate, utilizing a semantic map diffusion neural network, a diffusion representation of the semantic map; and
generate the infill semantic map from the diffusion representation.
12. The system of claim 8 , wherein the one or more processors are configured to cause the system to generate the modified digital image by:
generating a diffusion representation of the input digital image by utilizing diffusion layers of a digital image diffusion neural network; and
generating, utilizing denoising layers of the digital image diffusion neural network, the modified digital image from the diffusion representation and the infill semantic map.
13. A computer-implemented method comprising:
determining an infill modification indicating a region to fill for a digital image;
generating, utilizing a semantic map model, a semantic map from the digital image, wherein the semantic map comprises semantic classifications of pixels within the digital image;
generating, utilizing a generative semantic machine learning model, an infill semantic map from the digital image and the infill modification, wherein the infill semantic map comprises semantic classifications for the infill modification that indicates the region to fill for the digital image; and
generating, utilizing a generative image machine learning model, a modified digital image from the infill semantic map and the digital image.
14. The computer-implemented method of claim 13 , wherein generating the infill semantic map further comprises:
generating, utilizing the generative semantic machine learning model, the infill semantic map from the semantic map and the infill modification.
15. The computer-implemented method of claim 13 , further comprising determining the region to fill by generating, utilizing a segmentation neural network, a segmented digital image from the digital image.
16. The computer-implemented method of claim 13 , wherein determining the infill modification further comprises segmenting, utilizing a segmentation neural network, an object to remove from the digital image.
17. The computer-implemented method of claim 16 , further comprising generating the infill semantic map from the digital image to remove the object from the digital image by generating, utilizing the generative semantic machine learning model, semantic classifications for pixels within the region to be filled.
18. The computer-implemented method of claim 13 , wherein generating the infill semantic map comprises:
generating, utilizing diffusion layers of a semantic map diffusion neural network, a diffusion representation of the semantic map; and
generating, utilizing denoising layers of the semantic map diffusion neural network, the infill semantic map from the diffusion representation of the semantic map.
19. The computer-implemented method of claim 13 , wherein generating the infill semantic map comprises:
receiving a semantic editing input corresponding to the semantic map; and
generating the infill semantic map from the semantic map utilizing a semantic map diffusion neural network conditioned on the semantic editing input.
20. The computer-implemented method of claim 13 , wherein generating the modified digital image comprises:
generating a diffusion representation of the digital image by utilizing diffusion layers of a digital image diffusion neural network; and
generating, utilizing denoising layers of the digital image diffusion neural network, the modified digital image from the diffusion representation and the infill semantic map.