Generating light-source-specific parameters for digital images using a neural network

1. A non-transitory computer-readable medium storing computer-readable instructions which, when executed by at least one processing device, cause the at least one processing device to perform operations comprising:

extracting a latent feature vector from a digital image utilizing a first subset of common network layers of a source-specific-lighting-estimation-neural network;

extracting a common feature vector from the latent feature vector utilizing a second subset of common network layers of the source-specific-lighting-estimation-neural network; and

generating three-dimensional (“3D”) source-specific-lighting parameters based on the common feature vector utilizing parametric-specific-network layers of the source-specific-lighting-estimation-neural network by:

generating 3D-source-specific-distance parameters estimating one or more distances of one or more light sources from a reference point based on the latent feature vector utilizing distance-parametric-specific-network layers of the parametric-specific-network layers; and

generating source-specific-lighting parameters based on the common feature vector utilizing additional parametric-specific-network layers of the parametric-specific-network layers.

2. The non-transitory computer-readable medium of claim 1 , wherein generating the source-specific-lighting parameters comprises applying different sets of the additional parametric-specific-network layers to generate:

3D-source-specific-direction parameters estimating one or more directions of the one or more light sources with respect to the reference point;

source-specific-size parameters estimating one or more sizes of the one or more light sources; and

source-specific-color parameters estimating one or more colors of the one or more light sources.

3. The non-transitory computer-readable medium of claim 1 , wherein generating the 3D-source-specific-distance parameters comprises:

generating a first set of 3D-source-specific-distance parameters corresponding to a first predicted light source illuminating the digital image; and

generating a second set of 3D-source-specific-distance parameters corresponding to a second predicted light source illuminating the digital image.

4. The non-transitory computer-readable medium of claim 2 , wherein the first subset of common network layers comprises an encoder from a densely connected convolutional network, the second subset of common network layers comprises fully connected layers, and each set of the parametric-specific-network layers comprise a parametric-specific decoder.

5. The non-transitory computer-readable medium of claim 4 , wherein the operations further comprise generating the 3D-source-specific-distance parameters based on the latent feature vector and the 3D-source-specific-direction parameters utilizing distance-parametric-specific-network layers from the parametric-specific-network layers.

6. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise providing, for display on a computing device, a graphical user interface comprising:

a first set of lighting parameter controls for a first set of 3D-source-specific-lighting parameters corresponding to a first predicted light source illuminating a modified digital image; and

a second set of lighting parameter controls for a second set of 3D-source-specific-lighting parameters corresponding to a second predicted light source illuminating the modified digital image.

7. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise:

identifying a position-adjustment request to move a virtual object from a designated position within the digital image to a new designated position within the digital image;

adjusting a projection of the 3D-source-specific-lighting parameters for the new designated position within the digital image; and

based on the position-adjustment request, rendering an adjusted digital image comprising the virtual object at the new designated position illuminated according to the adjusted projection of the 3D-source-specific-lighting parameters.

8. The non-transitory computer-readable medium of claim 1 , wherein:

generating the 3D source-specific-lighting parameters comprises inferring the 3D-source-specific-lighting parameters directly from the common feature vector; and

the source-specific-lighting-estimation-neural network comprises less than ten million network parameters.

9. A system comprising:

one or more memory devices storing a digital image and a source-specific-lighting-estimation-neural network; and

one or more server devices that cause the system to:

extract a latent feature vector from the digital image utilizing a first subset of common network layers of the source-specific-lighting-estimation-neural network;

extract a common feature vector from the latent feature vector utilizing a second subset of common network layers of the source-specific-lighting-estimation-neural network; and

generate three-dimensional (“3D”) source-specific-lighting parameters based on the common feature vector utilizing parametric-specific-network layers of the source-specific-lighting-estimation-neural network by:

generating 3D-source-specific-distance parameters estimating one or more distances of one or more light sources from a reference point based on the latent feature vector utilizing distance-parametric-specific-network layers of the parametric-specific-network layers; and

generating source-specific-lighting parameters based on the common feature vector utilizing additional parametric-specific-network layers of the parametric-specific-network layers.

10. The system of claim 9 , wherein generating the source-specific-lighting parameters comprises applying different sets of the additional parametric-specific-network layers to generate:

3D-source-specific-direction parameters estimating one or more directions of the one or more light sources with respect to the reference point;

source-specific-size parameters estimating one or more sizes of the one or more light sources; and

source-specific-color parameters estimating one or more colors of the one or more light sources.

11. The system of claim 9 , wherein generating the 3D-source-specific-distance parameters comprises:

generating a first set of 3D-source-specific-distance parameters corresponding to a first predicted light source illuminating the digital image; and

generating a second set of 3D-source-specific-distance parameters corresponding to a second predicted light source illuminating the digital image.

12. The system of claim 9 , wherein the first subset of common network layers comprises an encoder from a densely connected convolutional network, the second subset of common network layers comprises fully connected layers, and each set of the parametric-specific-network layers comprise a parametric-specific decoder.

13. The system of claim 9 , wherein the one or more server devices further cause the system to:

identify a request to render a virtual object at a designated position within the digital image; and

based on the request, render a modified digital image comprising the virtual object at the designated position illuminated according to the 3D-source-specific-lighting parameters.

14. The system of claim 13 , wherein the one or more server devices further cause the system to:

provide, for display on a computing device, a graphical user interface comprising a set of lighting parameter controls for a set of 3D-source-specific-lighting parameters corresponding to a predicted light source illuminating the modified digital image; and

based on detecting a user interaction with a lighting parameter control from among the set of lighting parameter controls, adjust one or more 3D-source-specific-lighting parameters from the set of 3D-source-specific-lighting parameters corresponding to the predicted light source.

15. A method comprising:

extracting a latent feature vector from a digital image utilizing a first subset of common network layers of a source-specific-lighting-estimation-neural network;

extracting a common feature vector from the latent feature vector utilizing a second subset of common network layers of the source-specific-lighting-estimation-neural network; and

generating three-dimensional (“3D”) source-specific-lighting parameters based on the common feature vector utilizing parametric-specific-network layers of the source-specific-lighting-estimation-neural network by:

generating 3D-source-specific-distance parameters estimating one or more distances of one or more light sources from a reference point based on the latent feature vector utilizing distance-parametric-specific-network layers of the parametric-specific-network layers; and

generating source-specific-lighting parameters based on the common feature vector utilizing additional parametric-specific-network layers of the parametric-specific-network layers.

16. The method of claim 15 , further comprising training the source-specific-lighting-estimation-neural network in a first training stage by:

generating 3D-source-specific-predicted-lighting parameters based on a common-feature-training vector extracted from a digital training image utilizing the source-specific-lighting-estimation-neural network;

applying a differentiable-projection layer to the 3D-source-specific-predicted-lighting parameters to project a predicted environment map corresponding to the digital training image;

comparing the predicted environment map to a ground-truth-environment map corresponding to the digital training image to determine an environment-map loss; and

modifying internal parameters of the source-specific-lighting-estimation-neural network based on the environment-map loss.

17. The method of claim 16 , further comprising training the source-specific-lighting-estimation-neural network in a second training stage by:

generating subsequent 3D-source-specific-predicted-lighting parameters based on a subsequent common-feature-training vector extracted from a subsequent digital training image utilizing the source-specific-lighting-estimation-neural network;

comparing the subsequent 3D-source-specific-predicted-lighting parameters to ground-truth-source-specific-lighting parameters corresponding to the subsequent digital training image to determine lighting parameter losses; and

modifying internal parameters of the parametric-specific-network layers based on the lighting parameter losses while maintaining internal parameters of the first and second subsets of common network layers.

18. The method of claim 17 , wherein training the source-specific-lighting-estimation-neural network comprises:

adding the distance-parametric-specific-network layers to the parametric-specific-network layers for the second training stage; and

generating 3D-source-specific-predicted-distance parameters as part of the subsequent 3D-source-specific-predicted-lighting parameters, the 3D-source-specific-predicted-distance parameters estimating one or more distances of one or more predicted light sources from the reference point utilizing the distance-parametric-specific-network layers.

19. The method of claim 17 , wherein comparing the subsequent 3D-source-specific-predicted-lighting parameters to the ground-truth-source-specific-lighting parameters to determine the lighting parameter losses comprises:

comparing subsequent 3D-source-specific-predicted-distance parameters for a predicted light source to ground-truth-source-specific-distance parameters to determine a distance-parameter loss;

comparing subsequent 3D-source-specific-predicted-size parameters for the predicted light source to ground-truth-source-specific-size parameters to determine a size-parameter loss; and

comparing subsequent 3D-source-specific-predicted-color parameters for the predicted light source to ground-truth-source-specific-color parameters to determine a color-parameter loss.

20. The method of claim 15 , wherein generating the source-specific-lighting parameters comprises applying different sets of the additional parametric-specific-network layers to generate:

3D-source-specific-direction parameters estimating one or more directions of the one or more light sources with respect to the reference point;

source-specific-size parameters estimating one or more sizes of the one or more light sources; and

source-specific-color parameters estimating one or more colors of the one or more light sources.