Generating a modified digital image utilizing a human inpainting model
The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For example, in one or more embodiments the disclosed systems utilize generative machine learning models to create modified digital images portraying human subjects. In particular, the disclosed systems generate modified digital images by performing infill modifications to complete a digital image or human inpainting for portions of a digital image that portrays a human. Moreover, in some embodiments, the disclosed systems perform reposing of subjects portrayed within a digital image to generate modified digital images. In addition, the disclosed systems in some embodiments perform facial expression transfer and facial expression animations to generate modified digital images or animations.
1. A system comprising:
one or more memory devices comprising a digital image, a structure guidance map of a human, a parameter neural network, and a human inpainting generative adversarial neural network; and
one or more processors configured to cause the system to:
determine a human portrayed in the digital image and a region of the human to inpaint;
generate, utilizing an encoder, a structural encoding from the structure guidance map of the human;
generate, utilizing the encoder, a visual appearance encoding of the human;
generate, utilizing the parameter neural network, local appearance feature tensor from the visual appearance encoding; and
generate, utilizing a human inpainting generative adversarial neural network, a modified digital image comprising modified pixels of the region from the structural encoding and the local appearance feature tensor.
2. The system of claim 1 , wherein the one or more processors are configured to cause the system to generate a spatially varying scaling tensor and a spatially varying shifting tensor by utilizing the parameter neural network.
3. The system of claim 2 , wherein the one or more processors are configured to cause the system to modulate the structural encoding utilizing the spatially varying scaling tensor and the spatially varying shifting tensor.
4. The system of claim 1 , wherein the one or more processors are configured to cause the system to generate, utilizing a first style block of the human inpainting generative adversarial neural network, an intermediate feature vector by utilizing a modulation layer, a convolutional layer, and a demodulation layer.
5. The system of claim 1 , wherein the one or more processors are configured to cause the system to generate the structural encoding from at least one of a keypoint map, a segmentation map, or a pose map.
6. The system of claim 1 , wherein the one or more processors are configured to cause the system to train the human inpainting generative adversarial neural network by:
determining a partial reconstruction loss for a portion of the digital image that does not include the region of the human to inpaint; and
modifying parameters of the human inpainting generative adversarial neural network based on the partial reconstruction loss.
7. The system of claim 6 , wherein the one or more processors are configured to cause the system to train the human inpainting generative adversarial neural network by:
determining an adversarial loss for the region of the human to inpaint; and
modifying parameters of the human inpainting generative adversarial neural network based on the adversarial loss and the partial reconstruction loss.
8. The system of claim 1 , wherein the one or more processors are configured to cause the system to determine an indication of at least one of removing an object, expanding a digital image frame of the digital image, or removing an object from the region of the human.
9. A non-transitory computer-readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
determining a human portrayed in a digital image and a region of the human to inpaint;
generating, utilizing a first encoder, a structural encoding from a structure guidance map of a depiction of the human from the digital image;
generating, utilizing a second encoder, a visual appearance encoding from the depiction of the human from the digital image;
generating, utilizing a parameter neural network, local appearance feature tensor from the visual appearance encoding; and
generating, utilizing a human inpainting generative adversarial neural network, a modified digital image comprising modified pixels of the region from the structural encoding by modulating style blocks of the human inpainting generative adversarial neural network based on the local appearance feature tensor.
10. The non-transitory computer-readable medium of claim 9 , wherein generating the structural encoding and the visual appearance encoding further comprises generating, utilizing a hierarchical encoder, the structural encoding from the structure guidance map of the human and the visual appearance encoding from the human portrayed in the digital image, wherein the hierarchical encoder comprises a plurality of downsampling layers and upsampling layers of matching resolutions connected via skip connections.
11. The non-transitory computer-readable medium of claim 9 , wherein generating the modified digital image further comprises generating an intermediate feature vector for each style block of the human inpainting generative adversarial neural network by modulating the structural encoding utilizing a spatially varying scaling tensor and a spatially varying shifting tensor corresponding to a style block.
12. A computer-implemented method comprising:
determining, from a digital image, a depiction of a human and a region of the human to inpaint;
generating, utilizing an encoder, a structural encoding from a structure guidance map of the human;
generating, utilizing the encoder, a visual appearance encoding from the human portrayed in the digital image;
generating, utilizing a parameter neural network, local appearance feature tensor from the visual appearance encoding; and
generating, utilizing a human inpainting generative adversarial neural network, a modified digital image comprising modified pixels of the region from the structural encoding and the visual appearance encoding of the human.
13. The computer-implemented method of claim 12 , wherein generating the structural encoding and the visual appearance encoding further comprises generating, utilizing a hierarchical encoder comprising a plurality of downsampling layers and upsampling layers of matching resolutions connected via skip connections, the structural encoding from the structure guidance map of the human and the visual appearance encoding from the human portrayed in the digital image.
14. The computer-implemented method of claim 12 , wherein generating the local appearance feature tensor comprises generating, utilizing the parameter neural network, a spatially varying scaling tensor and a spatially varying shifting tensor.
15. The computer-implemented method of claim 12 , further comprising training the human inpainting generative adversarial neural network by:
determining a partial reconstruction loss for a portion of the digital image that does not include the region of the human to inpaint;
determining an adversarial loss for the region of the human to inpaint; and
modifying parameters of the human inpainting generative adversarial neural network based on the partial reconstruction loss and the adversarial loss.
16. The computer-implemented method of claim 14 , wherein generating the modified digital image further comprises generating, utilizing a first style block of the human inpainting generative adversarial neural network, an intermediate feature vector by modulating the structural encoding utilizing the spatially varying scaling tensor and the spatially varying shifting tensor.
17. The computer-implemented method of claim 16 , wherein generating the intermediate feature vector further comprises utilizing a modulation layer, a convolutional layer, and a demodulation layer to generate the intermediate feature vector.
18. The computer-implemented method of claim 16 , further comprising generating, utilizing a second style block of the human inpainting generative adversarial neural network, an additional intermediate feature vector from the intermediate feature vector, an additional spatially varying scaling tensor, and an additional spatially varying shifting tensor.
19. The computer-implemented method of claim 12 , wherein generating the structural encoding from the structure guidance map comprises generating the structural encoding from at least one of a keypoint map or a pose map.
20. The computer-implemented method of claim 12 , wherein generating the structural encoding from the structure guidance map comprises generating the structural encoding from at least one of a segmentation map or a digital image.