Generating and compositing hair pixels using generative neural networks
The present disclosure relates to systems, methods, and non-transitory computer readable media for generating and composting pixels of a digital image that depict hair of an individual using generative neural networks. In some embodiments, the disclosed systems receive a modification to a face crop enclosing a face depicted within a digital image. In some cases, the disclosed systems determine, from the modification, modified hair pixels within the face crop of the digital image and unmodified hair pixels outside of the face crop of the digital image. The disclosed systems generate, for the unmodified hair pixels outside of the face crop, replacement hair pixels that resemble the modified hair pixels utilizing a generative neural network. Additionally, the disclosed systems generate a modified digital image by replacing the unmodified hair pixels outside of the face crop with the replacement hair pixels.
1 . A method comprising:
identifying a modification to a face depicted within a face region of a digital image;
determining, from the modification utilizing a deep hair matte, modified hair pixels within the face region of the digital image and unmodified hair pixels outside of the face region of the digital image;
generating, utilizing a generative neural network for the unmodified hair pixels outside of the face region, replacement hair pixels that resemble the modified hair pixels; and
generating, utilizing an image compositing model, a modified digital image by replacing the unmodified hair pixels outside of the face region with the replacement hair pixels.
2 . The method of claim 1 , wherein:
determining the modified hair pixels within the face region comprises determining hair pixels affected by the modification to the face region; and
determining the unmodified hair pixels outside of the face region comprises determining hair pixels unaffected by the modification to the face region.
3 . The method of claim 1 , further comprising generating the deep hair matte from the digital image by utilizing a deep matting model to process image segmentations for the digital image.
4 . The method of claim 3 , further comprising:
generating a cropped deep hair mask by cropping the deep hair matte according to the face region of the digital image; and
wherein generating the replacement hair pixels comprises utilizing the generative neural network to generate pixels that resemble hair pixels indicated by the cropped deep hair mask.
5 . The method of claim 3 , wherein generating the deep hair matte comprises extracting, from the digital image, an instance segmentation map indicating one or more faces depicted within the digital image.
6 . The method of claim 5 , wherein generating the deep hair matte further comprises encoding, from the instance segmentation map, a part segmentation map defining hair pixels for the one or more faces depicted within the digital image.
7 . The method of claim 1 , wherein generating the modified digital image comprises utilizing a mesh warp to modify a resolution of the replacement hair pixels to match a resolution of the digital image.
8 . A system comprising:
a memory component; and
one or more processing devices coupled to the memory component, the one or more processing devices to perform operations comprising:
identifying a modification to a face depicted within a face region of a digital image;
determining, from the modification, modified hair pixels within the face region of the digital image and unmodified hair pixels outside of the face region of the digital image by generating a deep hair matte from a part segmentation map of the digital image;
generating, utilizing a generative neural network for the unmodified hair pixels outside of the face region, replacement hair pixels that resemble the modified hair pixels; and
generating, utilizing an image compositing model, a modified digital image by replacing the unmodified hair pixels outside of the face region with the replacement hair pixels.
9 . The system of claim 8 , wherein identifying the modification to the face depicted within the face region of the digital image comprises receiving an indication of applying a style modification neural network to pixels within the face region.
10 . The system of claim 8 , wherein generating the deep hair matte comprises:
extracting, from the digital image, an instance segmentation map indicating the face depicted within the digital image;
encoding, from the instance segmentation map, a part segmentation map defining hair pixels for the face depicted within the digital image; and
utilizing a deep matting model to generate the deep hair matte from the part segmentation map.
11 . The system of claim 8 , wherein generating the replacement hair pixels comprises utilizing the generative neural network to generate hair pixels indicated by the deep hair matte for replacing the unmodified hair pixels outside of the face region.
12 . The system of claim 8 , wherein generating the modified digital image comprises utilizing a mesh warp to align textures of the replacement hair pixels with textures of the digital image.
13 . The system of claim 8 , wherein:
determining the modified hair pixels within the face region comprises determining hair pixels affected by the modification to the face region; and
determining the unmodified hair pixels outside of the face region comprises determining hair pixels unaffected by the modification to the face region.
14 . The system of claim 8 , further comprising providing the modified digital image for display on a client device.
15 . A non-transitory computer readable medium storing instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
identifying a modification to a face depicted within a face region of a digital image;
determining, from the modification utilizing a deep hair matte, modified hair pixels within the face region of the digital image and unmodified hair pixels outside of the face region of the digital image;
generating, utilizing a generative neural network for the unmodified hair pixels outside of the face region, replacement hair pixels that resemble the modified hair pixels; and
generating, utilizing an image compositing model, a modified digital image by replacing the unmodified hair pixels outside of the face region with the replacement hair pixels.
16 . The non-transitory computer readable medium of claim 15 , wherein:
determining the modified hair pixels within the face region comprises determining hair pixels affected by the modification to the face region; and
determining the unmodified hair pixels outside of the face region comprises determining hair pixels unaffected by the modification to the face region.
17 . The non-transitory computer readable medium of claim 15 , further comprising generating the deep hair matte from the digital image by utilizing a deep matting model to process image segmentations for the digital image.
18 . The non-transitory computer readable medium of claim 17 , further comprising:
generating a cropped deep hair mask by cropping the deep hair matte according to the face region of the digital image; and
wherein generating the replacement hair pixels comprises utilizing the generative neural network to generate pixels that resemble hair pixels indicated by the cropped deep hair mask.
19 . The non-transitory computer readable medium of claim 17 , wherein generating the deep hair matte comprises extracting, from the digital image, an instance segmentation map indicating one or more faces depicted within the digital image.
20 . The non-transitory computer readable medium of claim 19 , wherein generating the deep hair matte further comprises encoding, from the instance segmentation map, a part segmentation map defining hair pixels for the one or more faces depicted within the digital image.