Editing shadows in digital images utilizing machine learning models
The present disclosure relates to systems, non-transitory computer-readable media, and methods for editing shadows in digital images. In particular, in some embodiments, the disclosed systems determine, utilizing a lighting estimation network, an environment map for a digital image, the environment map comprising a dominant light. In addition, in some embodiments, the disclosed systems generate, utilizing a lighting diffusion network, a diffused image from the digital image, the diffused image comprising smoothed shading. Moreover, in some embodiments, the disclosed systems generate, utilizing a shadow synthesis network, a shadowed image from the diffused image and a modified environment map comprising a modified dominant light. Furthermore, in some embodiments, the disclosed systems generate, from the diffused image and the shadowed image, a modified digital image comprising an edited shadow.
1 . A computer-implemented method comprising:
determining, utilizing a lighting estimation network, an environment map for a digital image, the environment map comprising a dominant light;
generating, utilizing a lighting diffusion network to remove at least one of specular highlights or hard shadows from the digital image, a diffused image from the digital image, the diffused image comprising smoothed shading and ambient lighting;
generating, utilizing a shadow synthesis network, a shadowed image from the diffused image and a modified environment map comprising a modified dominant light; and
generating, from the diffused image and the shadowed image, a modified digital image comprising an edited shadow.
2 . The computer-implemented method of claim 1 , wherein determining the environment map comprises determining at least one of a position, a size, or an intensity of the dominant light.
3 . The computer-implemented method of claim 1 , further comprising modeling the dominant light as a two-dimensional isotropic Gaussian light source.
4 . The computer-implemented method of claim 1 , wherein generating the diffused image comprises removing the hard shadows and the specular highlights of the digital image.
5 . The computer-implemented method of claim 1 , further comprising generating the modified environment map by changing at least a position, a size, or an intensity of the dominant light of the environment map.
6 . The computer-implemented method of claim 1 , wherein generating the shadowed image comprises applying the modified dominant light to the diffused image.
7 . The computer-implemented method of claim 1 , wherein generating the modified digital image comprises compositing the diffused image and the shadowed image as a weighted combination.
8 . A system comprising:
one or more memory devices; and
one or more processors coupled to the one or more memory devices that cause the system to perform operations comprising:
determining, utilizing a lighting estimation network, an environment map for a digital image, the environment map comprising a dominant light;
generating a diffused image comprising smoothed shading and ambient lighting from the digital image by utilizing a lighting diffusion network to remove hard shadows and specular highlights of the digital image;
generating a shadowed image from the diffused image and a modified environment map by utilizing a shadow synthesis network to apply a modified dominant light to the diffused image; and
generating a modified digital image comprising an edited shadow for the digital image by compositing the diffused image and the shadowed image.
9 . The system of claim 8 , wherein determining the environment map comprises determining a three-dimensional intensity of the dominant light.
10 . The system of claim 8 , wherein the one or more processors further cause the system to perform additional operations comprising:
providing, for display via a user interface of a client device, a lighting control element; and
determining, based on a user interaction with the lighting control element, at least one of a position, a size, or an intensity of the modified dominant light for the modified environment map.
11 . The system of claim 8 , wherein the one or more processors further cause the system to perform additional operations comprising:
determining the modified environment map comprising the modified dominant light and a new dominant light,
wherein generating the shadowed image from the diffused image and the modified environment map comprises utilizing the shadow synthesis network to apply the modified dominant light and the new dominant light to the diffused image.
12 . The system of claim 8 , wherein the one or more processors further cause the system to perform additional operations comprising:
determining a measure of lighting estimation loss based on a difference between the environment map and a ground truth environment map; and
modifying parameters of the lighting estimation network based on the measure of lighting estimation loss.
13 . The system of claim 8 , wherein the one or more processors further cause the system to perform additional operations comprising:
determining a measure of lighting diffusion loss based on a difference between the diffused image and a ground truth diffused image; and
modifying parameters of the lighting diffusion network based on the measure of lighting diffusion loss.
14 . The system of claim 8 , wherein the one or more processors further cause the system to perform additional operations comprising:
determining a measure of shadow synthesis loss based on a difference between the shadowed image and a ground truth shadowed image; and
modifying parameters of the shadow synthesis network based on the measure of shadow synthesis loss.
15 . A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
determining, utilizing a lighting estimation network, an environment map for a digital image, the environment map comprising a dominant light;
generating, utilizing a lighting diffusion network to remove at least one of specular highlights or hard shadows from the digital image, a diffused image from the digital image, the diffused image comprising smoothed shading and ambient lighting;
generating, utilizing a shadow synthesis network, a shadowed image from the diffused image and a modified environment map comprising a modified dominant light; and
generating, from the diffused image and the shadowed image, a modified digital image comprising an edited shadow.
16 . The non-transitory computer-readable medium of claim 15 , wherein determining the environment map for the digital image comprises converting nonparametric information for the dominant light into position, size, and intensity parameters for the dominant light.
17 . The non-transitory computer-readable medium of claim 15 , wherein:
generating the diffused image from the digital image comprises generating an ambient-lighted image, and
generating the shadowed image from the diffused image and the modified environment map comprises generating a dominant-lighted image.
18 . The non-transitory computer-readable medium of claim 15 , further storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to perform additional operations comprising:
generating a lighting data set that correlates facial shadows to parameters of dominant lights;
determining ground truth environment maps comprising the dominant lights of the lighting data set; and
modifying parameters of the lighting estimation network based on the ground truth environment maps.
19 . The non-transitory computer-readable medium of claim 15 , further storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to perform additional operations comprising:
receiving a user interaction with a lighting control element via a user interface of a client device; and
determining, based on the user interaction with the lighting control element, a change to at least one of a position, a size, or an intensity of the dominant light for the modified environment map.
20 . The non-transitory computer-readable medium of claim 15 , wherein the digital image comprises a portrait, and wherein generating the modified digital image comprises generating an updated portrait with the edited shadow.