Method and system for dynamically estimating light-source parameters from multiple images

1. A computer-implemented method for estimating lighting conditions of a scene captured from a plurality of digital images, the scene being illuminated by one or more light sources, the method comprising the steps of:

a) accessing the plurality of digital images of the scene, the plurality of images sharing substantially the same projection center;

b) generating, using a lighting-estimation neural network, lighting parameters of the scene comprising at least 3D-source-specific-lighting parameters, wherein the lighting-estimation neural network is configured for:

i. extracting from the plurality of images a corresponding number of latent feature vectors, using instances of a first subset of common network layers of the lighting-estimation neural network;

ii. transforming the latent feature vectors into common-coordinates latent feature vectors, using instances of a latent-rotation neural network, wherein the common coordinates are relative to the scene;

iii. merging the plurality of common-coordinates latent feature vectors into a single latent feature vector,

iv. extracting, from the single latent feature vector, a common-feature vector, using a second subset of the common network layers of the lighting-estimation neural network;

v. generating the lighting parameters of the scene, by inputting the common-feature vector through a plurality of parametric-specific-network layers of the light-estimating neural network, the 3D-source-specific-lighting parameters being associated with respective ones of the light sources detected by the lighting-estimation neural network.

2. The computer-implemented method of claim 1 , wherein:

step a) comprises receiving N digital images of the scene, each digital image having been captured at an azimuth angle ON, wherein one of said digital images is a reference image with a reference azimuth Oo;

step b) i) comprises inputting the N digital images into N instances of the first subset of common network layers, and generating therefrom N latent feature vectors, one of said feature vector being a reference latent feature vector;

step b) ii) comprises inputting N- 1 latent feature vectors and their associated azimuth angles relative to the reference azimuth Oo into respective instances of the latent-rotation neural network, whereby the transformed N- 1 latent feature vectors and the reference feature vector, result in N common-coordinates latent feature vectors; and

step b) iii) comprises inputting the N common-coordinate latent feature vectors into the pooling layer of the light-estimating neural network.

3. The computer-implemented method of claim 2 , wherein the azimuth angles ON of the digital images are estimated using a plurality of instances of an angle-estimation neural network.

4. The computer-implemented method of claim 3 , wherein the angle-estimation neural network comprises a slim angle-estimation network comprising fully-connected layers, and wherein estimating the azimuth angle ON of a given one of the digital images comprises inputting the corresponding latent feature vector of said image and the reference latent feature vector to the trained slim angle-estimation network.

5. The computer-implemented method of claim 3 , wherein the angle-estimation neural network comprises a full angle-estimation network based on a convolutional neural network (CNN) architecture with fully-connected layers, and wherein estimating the azimuth angle θ N of a given one of the digital images comprises inputting into the trained full angle-estimation neural network:

the corresponding latent feature vector of said image;

the reference latent feature vector associated to the reference image; and

said given one image and said reference image.

6. The computer-implemented method of claim 1 , wherein the first subset of the common network layers comprises an encoder neural network.

7. The computer-implemented method of claim 1 , wherein the latent-rotation neural network comprises at least one fully-connected layer of a size matching a size of the latent feature vectors.

8. The computer-implemented method of claim 1 , wherein the azimuth angles ON of the images are determined based on image information provided by the image-capture device used for capturing the digital images.

9. The computer-implemented method of claim 1 , wherein merging the plurality of common-coordinates feature vectors is performed by using a pooling layer of the light-estimating neural network.

10. The computer-implemented method of claim 1 , wherein the pooling layer is a maximum pooling layer or an average pooling layer, and comprises one or more fully-connected layers.

11. The computer-implemented method of claim 1 , wherein the parametric-specific-network layers generate, for each of the light sources detected by the lighting-estimation neural network, at least one of:

a. a 3D-source-specific-distance parameter estimating a distance of the light source from a reference point;

b. a 3D-source-specific-direction parameter estimating a direction of the light source with respect to the reference point;

c. a source-specific-size parameter estimating a size of the light source;

d. a source-specific-color or intensity parameter estimating a color of the light source.

12. The computer-implemented method of claim 1 , wherein the lighting parameters further comprise an ambient parameter representative of an overall illumination of the digital images when light sources are removed, and wherein the parametric-specific-network layers generate the ambient parameter.

13. The computer-implemented method of claim 1 , wherein the parametric-specific-network layers are fully-connected specific-network decoder layers, and wherein output sizes of the parametric-specific-network layers vary according to parameter type and a number of light sources detected.

14. The computer-implemented method of claim 1 , wherein the 3D-source-specific-direction, source-specific-size and source-specific-color parameters corresponding to any one of the light sources are generated based on the common-feature vector.

15. The computer-implemented method of claim 1 , wherein the 3D-source-specific distance parameter corresponding to one of the light sources is generated based on the common latent feature vector and the 3D-source-specific-direction parameter corresponding to the light source.

16. The computer-implemented method of claim 1 , further comprising a step of rendering at least one modified digital image comprising a virtual object illuminated according to the generated 3D-light parameters.

17. A system for estimating lighting conditions of a scene expressed as lighting parameters comprising at least three-dimensional (3D)-source-specific-lighting parameters, the system comprising:

a. a module for accessing a plurality of digital images of a scene sharing substantially the same projection center;

b. a trained lighting-estimation neural network for generating the 3D-source-specific-lighting parameters of the scene, the trained lighting-estimation neural network configured to:

i. extract a plurality of latent feature vectors, using a plurality of instances of a first subset of common network layers of the lighting-estimation neural network, corresponding to the plurality of digital images;

ii. transform, using a plurality of instances of a trained latent-rotation neural network, the plurality of latent feature vectors into common-coordinates latent feature vectors, wherein the common coordinates are relative to the scene;

iii. merge, using a pooling layer, the plurality of common-coordinates latent feature vectors into a single latent feature vector;

iv. extract, using a second subset of the common network layers of the lighting-estimation neural, a common feature vector from the single latent feature vector;

V. generate, using a plurality of parametric-specific-network layers, the 3D-source-specific-lighting parameters of the scene for each of light sources predicted by the lighting-estimation neural network.

18. The system of claim 17 , wherein the lighting parameters further comprise an ambient parameter, and wherein step v. comprises generating the ambient parameter.

19. The system of claim 17 , further comprising a rendering module, to render at least one modified image of the plurality of digital images, the modified image comprising a virtual object at a given position in said image, the virtual object being illuminated according to the lighting parameters.

20. The system of claim 17 , further comprising an angle-estimation module for estimating the azimuth angles θ N of the images using a plurality of instances of angle-estimation networks.

21. A computer-implemented method for estimating lighting conditions of a scene captured from a plurality of digital images, the scene being illuminated by one or more light sources, the method comprising:

generating 3D-source-specific-lighting parameters of the scene using a lighting-estimation neural network configured for:

extracting from the plurality of images a corresponding number of latent feature vectors;

transforming the latent feature vectors into common-coordinates latent feature vectors;

merging the plurality of common-coordinates latent feature vectors into a single latent feature vector;

extracting, from the single latent feature vector, 3D-source-specific-lighting parameters of the scene.