Estimating HDR lighting conditions from a single LDR digital image
The present disclosure is directed toward systems and methods for predicting lighting conditions. In particular, the systems and methods described herein analyze a single low-dynamic range digital image to estimate a set of high-dynamic range lighting conditions associated with the single low-dynamic range lighting digital image. Additionally, the systems and methods described herein train a convolutional neural network to extrapolate lighting conditions from a digital image. The systems and methods also augment low-dynamic range information from the single low-dynamic range digital image by using a sky model algorithm to predict high-dynamic range lighting conditions.
1. A computer-implemented method of estimating illumination from digital images, the computer-implemented method comprising:
estimating one or more lighting parameters from a single low-dynamic range digital image utilizing a neural network;
estimating one or more camera parameters from the single low-dynamic range digital image utilizing the neural network;
generating, based on the one or more lighting parameters, high-dynamic range lighting conditions for the single low-dynamic range digital image; and
generating a modified digital image from the single low-dynamic range digital image using the high-dynamic range lighting conditions and the one or more camera parameters.
2. The computer-implemented method as recited in claim 1 , wherein estimating the one or more lighting parameters from the single low-dynamic range digital image comprises estimating a sun position for the single low-dynamic range digital image.
3. The computer-implemented method as recited in claim 2 , wherein estimating the sun position for the single low-dynamic range digital image comprises generating a probability distribution over potential sun positions.
4. The computer-implemented method as recited in claim 2 , wherein the single low-dynamic range digital image is devoid of a captured sun.
5. The computer-implemented method as recited in claim 2 , wherein estimating the one or more lighting parameters from the single low-dynamic range digital image comprises estimating an atmospheric turbidity and an exposure.
6. The computer-implemented method as recited in claim 1 , wherein generating the modified digital image from the single low-dynamic range digital image using the high-dynamic range lighting conditions and the one or more camera parameters comprises:
calibrating a virtual camera with the one or more camera parameters; and
inserting an object into the modified digital image using the calibrated virtual camera.
7. The computer-implemented method as recited in claim 1 , wherein generating, based on the one or more lighting parameters, the high-dynamic range lighting conditions comprises generating a high-dynamic range outdoor environment map.
8. The computer-implemented method as recited in claim 1 , wherein generating the modified digital image from the single low-dynamic range digital image using the high dynamic range lighting conditions comprises inserting an object into the single low-dynamic range digital image, the object being illuminated according to the high-dynamic range lighting conditions.
9. The computer-implemented method as recited in claim 1 , wherein estimating the one or more camera parameters from the single low-dynamic range digital image utilizing the neural network comprises estimating geometric and radiometric camera calibration.
10. A system for estimating illumination from digital images, comprising:
a memory device comprising a neural network with a set of convolutional layers, a first head, and a second head; and
a computing device configured to cause the system to:
estimate a sun position for a single low-dynamic range digital image utilizing the set of convolutional layers and the first head of the neural network;
estimate sky parameters, independent from estimating the sun position, from the single low-dynamic range digital image utilizing the set of convolutional layers and the second head of the neural network; and
generate, based on the sun position and the sky parameters, high-dynamic range lighting conditions for the single low-dynamic range digital image.
11. The system as recited in claim 10 , wherein the computing device is configured to cause the system to generate, based on the sun position and the sky parameters, the high-dynamic range lighting conditions by utilizing a sky model to construct an environment map.
12. The system as recited in claim 11 , wherein utilizing the sky model to construct the environment map comprises creating a hypothetical horizon image depicting lighting conditions.
13. The system as recited in claim 10 , wherein the computing device is configured to cause the system to estimate the sky parameters by estimating an exposure and a turbidity.
14. The system as recited in claim 10 , wherein:
the single low-dynamic range digital image is devoid of a captured sun; and
the computing device is configured to estimate the sun position by generating a probability distribution over potential sun positions.
15. The system as recited in claim 10 , wherein the computing device is further configured to cause the system to:
extract a feature representation that captures illumination information of the single low-dynamic range digital image from a layer of the set of convolutional layers; and
use the feature representation to search for additional digital images with similar illumination to the single low-dynamic range digital image by comparing the feature representation to feature representations of the additional digital images.
16. The system as recited in claim 10 , wherein the computing device is further configured to cause the system to generate a modified digital image from the single low-dynamic range digital image using the high-dynamic range lighting conditions.
17. A non-transitory computer readable medium comprising instructions that, when executed by a processor, cause a computing device to:
receive a request to modify a single low-dynamic range digital image;
estimate a set of lighting parameters from the single low-dynamic range digital image utilizing a neural network;
generate, based on the set of lighting parameters, high-dynamic range lighting conditions for the single low-dynamic range digital image; and
generate a modified digital image from the single low-dynamic range digital image using the high-dynamic range lighting conditions.
18. The non-transitory computer readable medium of claim 17 , wherein the instructions that, when executed by the processor, cause the computing device to generate the high-dynamic range lighting conditions for the single low-dynamic range digital image by utilizing a sky model to construct an environment map based on the set of lighting parameters.
19. The non-transitory computer readable medium of claim 17 , further comprising instructions that, when executed by the processor, cause the computing device to estimate a set of camera parameters associated with an image capturing device used to capture the single low-dynamic range digital image.
20. The non-transitory computer readable medium of claim 19 , wherein the instructions that, when executed by the processor, cause the computing device to generate the modified digital image from the single low-dynamic range digital image by:
calibrating a virtual camera with the set of camera parameters;
inserting an object into the modified digital image using the calibrated virtual camera; and
lighting the object utilizing the high-dynamic range lighting conditions.