Deep-learning-based automatic skin retouching
Embodiments disclosed herein involve techniques for automatically retouching photos. A neural network is trained to generate a skin quality map from an input photo. The input photo is separated into high and low frequency layers which are separately processed. A high frequency path automatically retouches the high frequency layer using a neural network that accepts the skin quality map as an input. A low frequency path automatically retouches the low frequency layer using a color transformation generated by a second neural network and the skin quality map. The retouched high and low frequency layers are combined to generate the final output. In some embodiments, a training set for any or all of the networks is enhanced by applying a modification to an original image from a pair of retouched photos in the training set to improve the resulting performance of trained networks over different input conditions.
1. One or more non-transitory computer storage media storing computer-useable instructions that, when used by one or more computing devices, cause the one or more computing devices to perform operations comprising:
automatically generating a skin quality map from a skin mask of an input image, wherein the skin mask identifies regions of skin in the input image, and wherein the skin quality map is a probability map comprising a probability for each analyzed region that the analyzed region needs retouching;
separating the skin mask into a high frequency layer and a low frequency layer;
providing the skin quality map as an input into a first neural network configured to perform texture synthesis on the high frequency layer to generate a retouched high frequency layer;
applying a color transformation to the low frequency layer to generate a retouched low frequency layer, wherein parameters of the color transformation are generated using a second neural network and upsampled using the skin quality map; and
combining the retouched high frequency layer and the retouched low frequency layer.
2. The non-transitory media of claim 1 , wherein the skin mask is generated by removing non-skin regions from the input image, leaving only the regions of skin.
3. The non-transitory media of claim 1 , wherein the skin quality map is generated using a neural network trained using the same training set as the first neural network.
4. The non-transitory media of claim 1 , wherein the first neural network is a conditional generative adversarial network comprising a dilated residual network.
5. The non-transitory media of claim 1 , wherein the first neural network operates on patches of the high frequency layer.
6. The non-transitory media of claim 1 , wherein the operations additionally comprise enhancing a training set for at least one of the first or second neural networks.
7. The non-transitory media of claim 6 , wherein enhancing the training set comprises applying a modification to an original image from the training set before training.
8. The non-transitory media of claim 7 , wherein the modification comprises applying relighting to the original image.
9. The non-transitory media of claim 7 , wherein the modification comprises applying synthetic blemishes to the original image.
10. The non-transitory media of claim 7 , wherein the modification comprises applying palette-based photo recoloring to the original image.
11. A computerized method for automatically retouching an input image, the method comprising:
automatically generating a skin quality map from a skin mask of an input image, wherein the skin mask identifies regions of skin in the input image, and wherein the skin quality map is a probability map comprising a probability for each analyzed region that the analyzed region needs retouching;
separating the skin mask into a high frequency layer and a low frequency layer;
providing the skin quality map as an input into a first neural network configured to perform texture synthesis on the high frequency layer to generate a retouched high frequency layer;
applying a color transformation to the low frequency layer to generate a retouched low frequency layer, wherein parameters of the color transformation are generated using a second neural network and upsampled using the image quality map; and
combining the retouched high frequency layer and the retouched low frequency layer.
12. The method of claim 11 , wherein the first neural network is a conditional generative adversarial network.
13. The method of claim 11 , wherein the first neural network operates on patches of the high frequency layer.
14. The method of claim 11 , wherein the method additionally comprises enhancing a training set for at least one of the first or second neural networks by applying a modification to an original image from the training set before training.
15. The method of claim 14 , wherein the modification comprises at least one of applying relighting to the original image, applying synthetic blemishes to the original image, or applying palette-based photo recoloring to the original image.
16. A computer system comprising:
one or more hardware processors and memory configured to provide computer program instructions to the one or more hardware processors;
a skin quality detection network configured to generate a skin quality map from a skin mask of an input image, wherein the skin mask identifies regions of skin in the input image, and wherein the skin quality map is a probability map comprising a probability for each analyzed region that the analyzed region needs retouching;
a frequency separator configured to separate the skin mask into a high frequency layer and a low frequency layer;
a means for performing texture synthesis on the high frequency layer by providing the skin quality map as an input into a first neural network configured to generate a retouched high frequency layer;
a means for applying a color transformation to the low frequency layer to generate a retouched low frequency layer using parameters of the color transformation generated using a second neural network and upsampled using the skin quality map; and
a combiner configured to reconstitute the retouched high frequency layer and the retouched low frequency layer.
17. The computer system of claim 16 , wherein the first neural network is a conditional generative adversarial network comprising a dilated residual network.
18. The computer system of claim 16 , wherein the first neural network operates on patches of the high frequency layer.
19. The computer system of claim 16 , at least one of the first or second neural networks being trained by a training set, the training set being enhanced by a modification to an original image from the training set before training.
20. The computer system of claim 19 , wherein the modification comprises at least one of an application of relighting to the original image, an application of synthetic blemishes to the original image, or an application of palette-based photo recoloring to the original image.