Utilizing a warped digital image with a reposing model to synthesize a modified digital image
The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For example, in one or more embodiments the disclosed systems utilize generative machine learning models to create modified digital images portraying human subjects. In particular, the disclosed systems generate modified digital images by performing infill modifications to complete a digital image or human inpainting for portions of a digital image that portrays a human. Moreover, in some embodiments, the disclosed systems perform reposing of subjects portrayed within a digital image to generate modified digital images. In addition, the disclosed systems in some embodiments perform facial expression transfer and facial expression animations to generate modified digital images or animations.
1 . A computer-implemented method comprising:
generating a texture map from pixel values of a source digital image that depicts a human and from a pose map of the source digital image;
generating a warped digital image by warping the texture map with a target pose map;
combining the warped digital image and the target pose map to generate, utilizing a pose encoder, a warped pose feature map;
generating, utilizing a texture map appearance encoder, a global texture map appearance vector from the texture map; and
synthesizing, utilizing a reposing generative adversarial neural network, a modified digital image that depicts the human according to the target pose map based on the warped pose feature map generated from combining the warped digital image and the target pose map and the global texture map appearance vector generated from the texture map.
2 . The computer-implemented method of claim 1 , wherein generating the texture map further comprises:
generating, utilizing a three-dimensional model, the pose map from the source digital image; and
generating the texture map by projecting the pixel values of the source digital image guided by the pose map of the source digital image.
3 . The computer-implemented method of claim 1 , wherein generating the warped digital image further comprises:
determining UV coordinates from the target pose map; and
generating the warped digital image by rearranging the pixel values from the source digital image in the texture map based on the UV coordinates from the target pose map.
4 . The computer-implemented method of claim 1 , further comprising:
generating, utilizing an image appearance encoder of the reposing generative adversarial neural network, a local appearance vector from the warped digital image;
generating, utilizing a parameter neural network, a local appearance feature tensor from the local appearance vector; and
synthesizing, utilizing the reposing generative adversarial neural network, the modified digital image that depicts the human according to the target pose map based on the warped pose feature map, the global texture map appearance vector, and the local appearance feature tensor.
5 . The computer-implemented method of claim 4 , further comprising generating a globally modified local appearance feature tensor by combining the global texture map appearance vector and the local appearance feature tensor.
6 . The computer-implemented method of claim 5 , further comprising generating, utilizing a style block of the reposing generative adversarial neural network, an intermediate feature vector by modulating the warped pose feature map utilizing the globally modified local appearance feature tensor.
7 . The computer-implemented method of claim 1 , wherein:
generating the warped digital image further comprises generating, utilizing a coordinate inpainting generative neural network, an inpainted warped digital image; and
generating the warped pose feature map comprises generating, utilizing the pose encoder, the warped pose feature map from the inpainted warped digital image.
8 . The computer-implemented method of claim 1 , further comprising training the reposing generative adversarial neural network by:
generating a body mask for the human depicted in the modified digital image; and
determining a measure of loss for a portion of the modified digital image within the body mask utilizing a first loss weight.
9 . The computer-implemented method of claim 8 , further comprising training the reposing generative adversarial neural network by:
determining an additional measure of loss for an additional portion of the modified digital image outside the body mask utilizing a second loss weight; and
modifying parameters of the reposing generative adversarial neural network based on the measure of loss and the additional measure of loss.
10 . A system comprising:
one or more memory devices comprising a source digital image, a target pose map, and a reposing neural network; and
one or more processors configured to cause the system to:
generate a texture map from pixel values of the source digital image that depicts a human and a pose map of the source digital image;
generate a warped digital image by warping the texture map with the target pose map;
combine the warped digital image and the target pose map to generate, utilizing a pose encoder, a warped pose feature map; and
synthesize, utilizing the reposing neural network, a modified digital image that depicts the human according to the target pose map and the warped pose feature map generated from combining the warped digital image and the target pose map.
11 . The system of claim 10 , wherein the one or more processors are configured to cause the system to:
generate, utilizing a three-dimensional model, the pose map from the source digital image;
generate the texture map by projecting the pixel values of the source digital image guided by the pose map of the source digital image; and
generate the warped digital image by rearranging the pixel values from the source digital image in the texture map with UV coordinates from the target pose map.
12 . The system of claim 10 , wherein the one or more processors are configured to cause the system to:
generate, utilizing a coordinate inpainting generative neural network, an inpainted warped digital image from the warped digital image; and
generate, utilizing the pose encoder, the warped pose feature map from the inpainted warped digital image.
13 . The system of claim 10 , wherein the one or more processors are configured to cause the system to:
generate, utilizing a texture map appearance encoder, a global texture map appearance vector from the texture map; and
generate, utilizing an image appearance encoder, a warped image feature vector from the warped digital image.
14 . The system of claim 13 , wherein the one or more processors are configured to cause the system to:
generate, utilizing a parameter neural network, a local appearance feature tensor from the warped image feature vector; and
generate a globally modified local appearance feature tensor by combining the global texture map appearance vector and the local appearance feature tensor.
15 . The system of claim 14 , wherein the one or more processors are configured to cause the system to synthesize the modified digital image that depicts the human according to the target pose map utilizing the reposing neural network from the warped pose feature map and the globally modified local appearance feature tensor.
16 . A non-transitory computer-readable medium storing executable instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
generating a warped digital image by warping a texture map of a digital image with a target pose map;
combining the warped digital image and the target pose map to generate, utilizing a pose encoder, a warped pose feature map;
generating, utilizing a parameter neural network, local appearance feature tensor from the warped digital image; and
synthesizing, utilizing a reposing generative adversarial neural network, a modified digital image that depicts a human according to the target pose map based on the warped pose feature map generated from combining the warped digital image and the target pose map and the local appearance feature tensor.
17 . The non-transitory computer-readable medium of claim 16 , the operations further comprising generating the warped digital image by rearranging pixel values from the digital image in the texture map with UV coordinates from the target pose map.
18 . The non-transitory computer-readable medium of claim 16 , the operations further comprising:
generating a globally modified local appearance feature tensor from the local appearance feature tensor and a global texture map appearance vector from the texture map; and
modulating the warped pose feature map via the reposing generative adversarial neural network utilizing the globally modified local appearance feature tensor.
19 . The non-transitory computer-readable medium of claim 16 , the operations further comprising generating, utilizing a coordinate inpainting generative neural network, an inpainted warped digital image.
20 . The non-transitory computer-readable medium of claim 19 , the operations further comprising generating, utilizing the pose encoder, the warped pose feature map from the inpainted warped digital image.