Synthesizing a modified digital image utilizing a reposing model
The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For example, in one or more embodiments the disclosed systems utilize generative machine learning models to create modified digital images portraying human subjects. In particular, the disclosed systems generate modified digital images by performing infill modifications to complete a digital image or human inpainting for portions of a digital image that portrays a human. Moreover, in some embodiments, the disclosed systems perform reposing of subjects portrayed within a digital image to generate modified digital images. In addition, the disclosed systems in some embodiments perform facial expression transfer and facial expression animations to generate modified digital images or animations.
1 . A computer-implemented method comprising:
generating, utilizing a pose encoder, a pose feature map from a target pose map and a source digital image that depicts a human;
generating, utilizing a texture map appearance encoder, a global texture map appearance vector from a texture map of the source digital image;
generating, utilizing a parameter neural network, a local appearance feature tensor from the source digital image; and
synthesizing, utilizing a reposing generative adversarial neural network, a modified digital image that depicts the human according to the target pose map based on the pose feature map, the local appearance feature tensor, and the global texture map appearance vector.
2 . The computer-implemented method of claim 1 , wherein generating the pose feature map, utilizing the pose encoder comprises generating the pose feature map utilizing a hierarchical pose encoder by:
generating, utilizing a downsampling layer of the hierarchical pose encoder, an intermediate downsampled feature vector having a resolution;
generating utilizing an upsampling layer of the hierarchical pose encoder, an intermediate upsampled feature vector having the resolution; and
generating, via a skip connection, a combined feature vector from the intermediate downsampled feature vector and the intermediate upsampled feature vector.
3 . The computer-implemented method of claim 2 , wherein generating the pose feature map utilizing the hierarchical pose encoder further comprises:
generating, utilizing an additional downsampling layer of the hierarchical pose encoder, an additional intermediate downsampled feature vector having an additional resolution;
generating utilizing an additional upsampling layer of the hierarchical pose encoder from the combined feature vector, an additional intermediate upsampled feature vector having the additional resolution; and
combining, via a skip connection, the additional intermediate downsampled feature vector and the additional intermediate upsampled feature vector to generate the pose feature map.
4 . The computer-implemented method of claim 1 , wherein generating the global texture map appearance vector from the texture map of the source digital image comprises generating a one-dimensional appearance vector utilizing a hierarchical appearance encoder.
5 . The computer-implemented method of claim 1 , wherein generating the local appearance feature tensor further comprises:
generating, utilizing an image appearance encoder, a local appearance vector from the source digital image; and
generating, utilizing the parameter neural network, the local appearance feature tensor from the local appearance vector, the local appearance feature tensor comprising a spatially varying scaling tensor and a spatially varying shifting tensor.
6 . The computer-implemented method of claim 1 , wherein synthesizing the modified digital image utilizing the reposing generative adversarial neural network comprises generating the modified digital image from the pose feature map by:
generating globally modified local appearance feature tensor by combining the local appearance feature tensor and the global texture map appearance vector; and
modulating the reposing generative adversarial neural network utilizing the globally modified local appearance feature tensor.
7 . The computer-implemented method of claim 1 , further comprising training the reposing generative adversarial neural network by:
generating incomplete digital images from an unpaired image dataset by applying masks to a plurality of source digital images portraying humans in the unpaired image dataset; and
synthesizing, utilizing the reposing generative adversarial neural network, a plurality of inpainted digital images from the incomplete digital images and target pose maps of the plurality of source digital images.
8 . The computer-implemented method of claim 7 , further comprising modifying parameters of the reposing generative adversarial neural network based on measures of loss between the plurality of inpainted digital images and the plurality of source digital images.
9 . The computer-implemented method of claim 1 , further comprising training the reposing generative adversarial neural network utilizing an unpaired dataset comprising the source digital image by:
determining, utilizing a discriminator neural network, an adversarial loss from the modified digital image; and
modifying parameters of the reposing generative adversarial neural network based on the adversarial loss from the modified digital image generated from the unpaired dataset.
10 . A system comprising:
one or more memory devices comprising a source digital image that depicts a human, a texture map of the source digital image, a target pose map of the human, and a reposing neural network; and
one or more processors configured to cause the system to:
generate, utilizing a texture map appearance encoder of the reposing neural network, a global texture map appearance vector from the texture map;
generate, utilizing an image appearance encoder of the reposing neural network, a local appearance vector from the source digital image;
generate, utilizing a parameter neural network, local appearance feature tensor from the local appearance vector; and
synthesize, utilizing the reposing neural network, a modified digital image that depicts the human according to the target pose map based on the source digital image, the local appearance feature tensor, and the global texture map appearance vector.
11 . The system of claim 10 , wherein the one or more processors are configured to cause the system to:
generate the global texture map appearance vector from the texture map of the source digital image comprises generating a one-dimensional appearance vector utilizing a hierarchical appearance encoder; and
generate the local appearance feature tensor from the local appearance vector comprises a spatially varying scaling tensor and a spatially varying shifting tensor.
12 . The system of claim 10 , wherein the one or more processors are configured to cause the system to generate globally modified local appearance feature tensor by combining the global texture map appearance vector with the local appearance feature tensor.
13 . The system of claim 12 , wherein the one or more processors are configured to cause the system to modulate the reposing neural network utilizing the globally modified local appearance feature tensor.
14 . The system of claim 10 , wherein the one or more processors are configured to cause the system to generate the texture map from pixel values of the source digital image and a pose map of the source digital image.
15 . The system of claim 10 , wherein the one or more processors are configured to cause the system to:
train the reposing neural network by:
generating incomplete digital images from an unpaired image dataset by applying masks to a plurality of source digital images portraying humans in the unpaired image dataset;
synthesizing, utilizing the reposing neural network, a plurality of inpainted digital images from the incomplete digital images and target pose maps of the plurality of source digital images; and
modifying parameters of the reposing neural network based on measures of loss between the plurality of inpainted digital images and the plurality of source digital images.
16 . A non-transitory computer-readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
generating, utilizing a texture map appearance encoder of a reposing neural network, a global texture map appearance vector from a texture map;
generating, utilizing an image appearance encoder of the reposing neural network, a local appearance vector from a source digital image that depicts a human;
generating, a combined vector from the global texture map appearance vector and the local appearance vector; and
synthesizing, utilizing a reposing generative adversarial neural network, a modified digital image that depicts the human according to a target pose map based on the combined vector.
17 . The non-transitory computer-readable medium of claim 16 , the operations further comprising:
generating the texture map from pixel values of the source digital image and a pose map of the source digital image; and
generating the global texture map appearance vector from the texture map of the source digital image comprises generating a one-dimensional appearance vector utilizing a hierarchical appearance encoder.
18 . The non-transitory computer-readable medium of claim 16 , the operations further comprising:
generating a local appearance feature tensor from the local appearance vector, the local appearance feature tensor comprises a spatially varying scaling tensor and a spatially varying shifting tensor; and
generating the combined vector by combining the global texture map appearance vector with the local appearance feature tensor.
19 . The non-transitory computer-readable medium of claim 16 , the operations further comprising synthesizing the modified digital image by modulating a style block of the reposing neural network utilizing the combined vector.
20 . The non-transitory computer-readable medium of claim 16 , the operations further comprising training the reposing neural network by modifying parameters of the reposing neural network based on an adversarial loss from the modified digital image generated from an unpaired dataset comprising the source digital image.