Dynamically estimating light-source-specific parameters for digital images using a neural network
This disclosure relates to methods, non-transitory computer readable media, and systems that can render a virtual object in a digital image by using a source-specific-lighting-estimation-neural network to generate three-dimensional (“3D”) lighting parameters specific to a light source illuminating the digital image. To generate such source-specific-lighting parameters, for instance, the disclosed systems utilize a compact source-specific-lighting-estimation-neural network comprising both common network layers and network layers specific to different lighting parameters. In some embodiments, the disclosed systems further train such a source-specific-lighting-estimation-neural network to accurately estimate spatially varying lighting in a digital image based on comparisons of predicted environment maps from a differentiable-projection layer with ground-truth-environment maps.
1. A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause a computing device to:
identify a request to render a virtual object at a designated position within a digital image;
extract a common feature vector from the digital image utilizing common network layers of a source-specific-lighting-estimation-neural network;
generate, utilizing separate parallel parametric-specific-network layers, a plurality of different three-dimensional (“3D”) source-specific-lighting parameters of a light source illuminating the digital image comprising a first 3D source-specific-lighting parameter of the light source and a second 3D source-specific-lighting parameter of the light source by:
generating the first 3D source-specific-lighting parameter of the light source by processing the common feature vector from the common network layers utilizing a first parametric-specific-network layer of the source-specific-lighting-estimation-neural network; and
generating the second 3D source-specific-lighting parameter of the light source by processing the common feature vector from the common network layers utilizing a second parametric-specific-network layer of the source-specific-lighting-estimation-neural network, the second parametric-specific-network layer being different than the first parametric-specific-network layer and in parallel with the first parametric-specific-network layer; and
based on the request, render a modified digital image comprising the virtual object at the designated position illuminated according to the plurality of different 3D-source-specific-lighting parameters of the light source.
2. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to generate the plurality of different 3D-source-specific-lighting parameters by:
generating, utilizing the separate parallel parametric-specific-network layers, a first set of different 3D-source-specific-lighting parameters corresponding to a first predicted light source illuminating the digital image; and
generating, utilizing the separate parallel parametric-specific-network layers, a second set of different 3D-source-specific-lighting parameters corresponding to a second predicted light source illuminating the digital image.
3. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to provide, for display on the computing device, a graphical user interface comprising:
a first set of lighting parameter controls for a first set of different 3D-source-specific-lighting parameters corresponding to a first predicted light source illuminating the modified digital image; and
a second set of lighting parameter controls for a second set of different 3D-source-specific-lighting parameters corresponding to a second predicted light source illuminating the modified digital image.
4. The non-transitory computer readable medium of claim 1 , further comprising instructions that, when executed by the at least one processor, cause the computing device to:
identify a position-adjustment request to move the virtual object from the designated position within the digital image to a new designated position within the digital image;
adjust a projection of the plurality of different 3D-source-specific-lighting parameters for the new designated position within the digital image; and
based on the position-adjustment request, render an adjusted digital image comprising the virtual object at the new designated position illuminated according to the adjusted projection of the plurality of different 3D-source-specific-lighting parameters.
5. The non-transitory computer readable medium of claim 1 , wherein:
the instructions, when executed by the at least one processor, cause the computing device to generate the plurality of different 3D source-specific-lighting parameters by inferring the plurality of different 3D-source-specific-lighting parameters directly from the common feature vector; and
the source-specific-lighting-estimation-neural network comprises less than ten million network parameters.
6. In a digital medium environment for rendering digitally altered scenes, a computer-implemented method for estimating parametric lighting conditions, comprising:
identifying a request to render a virtual object at a designated position within a digital image;
extracting a common feature vector from the digital image utilizing common network layers of a source-specific-lighting-estimation-neural network;
generating, utilizing separate parallel parametric-specific-network layers, a plurality of different three-dimensional (“3D”) source-specific-lighting parameters of a light source illuminating the digital image comprising a first 3D source-specific-lighting parameter of the light source and a second 3D source-specific-lighting parameter of the light source by:
generating the first 3D source-specific-lighting parameter of the light source by processing the common feature vector from the common network layers utilizing a first parametric-specific-network layer of the source-specific-lighting-estimation-neural network; and
generating the second 3D source-specific-lighting parameter of the light source by processing the common feature vector from the common network layers utilizing a second parametric-specific-network layer of the source-specific-lighting-estimation-neural network, the second parametric-specific-network layer being different than the first parametric-specific-network layer and in parallel with the first parametric-specific-network layer; and
based on the request, rendering a modified digital image comprising the virtual object at the designated position illuminated according to the plurality of different 3D-source-specific-lighting parameters of the light source.
7. The computer-implemented method of claim 6 , further comprising:
receiving, from a computing device, the request to render the virtual object at the designated position within a two-dimensional image; and
based on receiving the request, rendering, within a graphical user interface of the computing device, a modified two-dimensional image comprising the virtual object at the designated position illuminated according to the plurality of different 3D-source-specific-lighting parameters.
8. A system comprising:
one or more memory devices storing a digital image and a source-specific-lighting-estimation-neural network; and
one or more server devices that cause the system to:
identify a request to render a virtual object at a designated position within the digital image;
extract a common feature vector from the digital image utilizing common network layers of a source-specific-lighting-estimation-neural network;
generate, utilizing separate parallel parametric-specific-network layers, a plurality of different three-dimensional (“3D”) source-specific-lighting parameters of a light source illuminating the digital image comprising a first 3D source-specific-lighting parameter of the light source and a second 3D source-specific-lighting parameter of the light source by:
generating the first 3D source-specific-lighting parameter of the light source by processing the common feature vector from the common network layers utilizing a first parametric-specific-network layer of the source-specific-lighting-estimation-neural network; and
generating the second 3D source-specific-lighting parameter of the light source by processing the common feature vector from the common network layers utilizing a second parametric-specific-network layer of the source-specific-lighting-estimation-neural network, the second parametric-specific-network layer being different than the first parametric-specific-network layer and in parallel with the first parametric-specific-network layer; and
based on the request, render a modified digital image comprising the virtual object at the designated position illuminated according to the plurality of different 3D-source-specific-lighting parameters of the light source.
9. The system of claim 8 , wherein the one or more server devices cause the system to generate the plurality of different 3D-source-specific-lighting parameters by:
generating, utilizing the separate parallel parametric-specific-network layers, a first set of different 3D-source-specific-lighting parameters corresponding to a first predicted light source illuminating the digital image; and
generating, utilizing the separate parallel parametric-specific-network layers, a second set of different 3D-source-specific-lighting parameters corresponding to a second predicted light source illuminating the digital image.
10. The system of claim 8 , wherein the one or more server devices cause the system to provide, for display on a computing device, a graphical user interface comprising:
a first set of lighting parameter controls for a first set of different 3D-source-specific-lighting parameters corresponding to a first predicted light source illuminating the modified digital image; and
a second set of lighting parameter controls for a second set of different 3D-source-specific-lighting parameters corresponding to a second predicted light source illuminating the modified digital image.
11. The system of claim 10 , wherein the one or more server devices further cause the system to:
detect a user interaction with a lighting parameter control from the first set of lighting parameter controls; and
in response to the user interaction, adjust one or more 3D-source-specific-lighting parameters from the first set of different 3D-source-specific-lighting parameters corresponding to the first predicted light source.