Artificial intelligence techniques for extrapolating HDR panoramas from LDR low FOV images
In some examples, a computing system accesses a field of view (FOV) image that has a field of view less than 360 degrees and has low dynamic range (LDR) values. The computing system estimates lighting parameters from a scene depicted in the FOV image and generates a lighting image based on the lighting parameters. The computing system further generates lighting features generated the lighting image and image features generated from the FOV image. These features are aggregated into aggregated features and a machine learning model is applied to the image features and the aggregated features to generate a panorama image having high dynamic range (HDR) values.
1 . A computer-implemented method in which one or more processing devices perform operations comprising:
accessing a field of view (FOV) image that has a field of view less than 360 degrees and has low dynamic range (LDR) values;
generating a lighting image based on lighting parameters estimated from a scene depicted in the FOV image;
generating, using a mapping module, a set of features from a random noise signal;
aggregating lighting features generated from the lighting image, image features generated from the FOV image, and the set of features generated from the random noise signal to generate aggregated features; and
applying a machine learning model to the image features and the aggregated features to generate a panorama image having high dynamic range (HDR) values.
2 . The computer-implemented method of claim 1 , further comprising updating the lighting parameters based on a user input, wherein the lighting image is generated based on updated lighting parameters.
3 . The computer-implemented method of claim 1 , wherein applying a machine learning model to the image features and the aggregated features to generate a panorama image having high dynamic range (HDR) values comprises:
applying a machine learning model to the image features and the aggregated features to generate a panorama image having low dynamic range (LDR) values; and
combining the lighting image with the panorama image having the LDR values to generate the high dynamic range (HDR) values.
4 . The computer-implemented method of claim 1 , wherein the lighting image is a spherical Gaussian image.
5 . The computer-implemented method of claim 1 , wherein the panorama image represents a field of view of 360 degrees.
6 . A system, comprising:
a memory component;
a processing device coupled to the memory component, the processing device to perform operations comprising:
generating a lighting image based on lighting parameters for a scene depicted in a field of view (FOV) image, the FOV image having a field of view less than 360 degrees and low dynamic range (LDR) values;
generating, using a mapping module, a set of features from a random noise signal;
aggregating the set of features, lighting features generated from the lighting image, and image features generated from the FOV image to generate aggregated features; and
generating a panorama image having high dynamic range (HDR) values based, at least in part, upon the image features and the aggregated features.
7 . The system of claim 6 , wherein the lighting parameters are specified by a user input or estimated from the scene and updated based on a user input.
8 . The system of claim 6 , wherein generate a panorama image having high dynamic range (HDR) values based, at least in part, upon the image features and the aggregated features comprises:
applying a machine learning model to the image features and the aggregated features to generate a panorama image having low dynamic range (LDR) values; and
combining the lighting image with the panorama image having the LDR values to generate the high dynamic range (HDR) values.
9 . The system of claim 8 , wherein the machine learning model is a generative adversarial network (GAN) model.
10 . The system of claim 6 , wherein the lighting image is a spherical Gaussian image.
11 . The system of claim 6 , wherein the panorama image represents a field of view of 360 degrees.
12 . A non-transitory computer-readable medium, storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:
accessing a field of view (FOV) image that has a field of view less than 360 degrees and has low dynamic range (LDR) values;
generating a lighting image based on lighting parameters estimated from a scene depicted in the FOV image;
a step for generating aggregated features based on the lighting image; and
a step for applying a generative adversarial network (GAN) model to image features generated from the FOV image and the aggregated features to generate a panorama image having high dynamic range (HDR) values.
13 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise: updating the lighting parameters based on a user input, wherein the lighting image is generated based on updated lighting parameters.
14 . The non-transitory computer-readable medium of claim 12 , wherein the operations further comprise: generating, using a mapping module, a set of features from a random noise signal, wherein the step for generating the aggregated features comprises aggregating lighting features generated from the lighting image, the image features, and the set of features.
15 . The non-transitory computer-readable medium of claim 12 , wherein the step for generating a panorama image having high dynamic range (HDR) values based on the image features and the aggregated features comprises:
combining the lighting image with the panorama image having the LDR values to generate the high dynamic range (HDR) values.
16 . The non-transitory computer-readable medium of claim 12 , wherein the lighting image is a spherical Gaussian image.
17 . The non-transitory computer-readable medium of claim 12 , wherein the panorama image represents a field of view of 360 degrees.