Multi-frame depth-based multi-camera relighting of images

1. A method comprising:

capturing a plurality of first images using a first image sensor;

capturing a plurality of second images using a second image sensor;

estimating depth information based on at least one of the first images and at least one of the second images;

obtaining information related to a lighting direction from an artificial intelligence (AI) light director, wherein the AI light director is trained to determine one or more lighting directions; and

generating at least one relit image using at least one of the first and second images based on the obtained information related to the lighting direction and the estimated depth information.

2. The method of claim 1 , wherein at least some of the first and second images are captured under different exposure conditions.

3. The method of claim 1 , wherein obtaining the information related to the lighting direction comprises:

receiving a user input that selects one relighting condition among a plurality of relighting conditions.

4. The method of claim 1 , wherein the first and second image sensors are located on or in a surface of an electronic device.

5. The method of claim 1 , wherein the information related to the lighting direction comprises an x-coordinate, a y-coordinate, a z-coordinate, a color, and an intensity for each of one or more light sources.

6. The method of claim 1 , wherein:

an approximate loss is determined based on comparing a raw image and a lighting direction applied to the raw image to an ideal relit image; and

the AI light director is trained using the approximated loss.

7. The method of claim 1 , wherein:

an AI-directed relit image is generated using the lighting direction on a raw image in a multi-frame depth-based relighting engine;

an actual loss is determined based on comparing the AI-directed relit image to an ideal relit image; and

the AI light director is trained using the actual loss.

8. An electronic device comprising:

a first image sensor configured to capture a plurality of first images;

a second image sensor configured to capture a plurality of second images; and

at least one processor operably coupled to the first and second image sensors, the at least one processor configured to:

estimate depth information based on at least one of the first images and at least one of the second images;

obtain information related to a lighting direction from an artificial intelligence (AI) light director, wherein the AI light director is trained to determine one or more lighting directions; and

generate at least one relit image using at least one of the first and second images based on the obtained information related to the lighting direction and the estimated depth information.

9. The electronic device of claim 8 , wherein the first and second image sensors are configured to capture at least some of the first and second images under different exposure conditions.

10. The electronic device of claim 8 , wherein, to obtain the information related to the lighting direction, the at least one processor is configured to receive a user input that selects one relighting condition among a plurality of relighting conditions.

11. The electronic device of claim 8 , wherein the first and second image sensors are located on or in a surface of the electronic device.

12. The electronic device of claim 8 , wherein the information related to the lighting direction comprises an x-coordinate, a y-coordinate, a z-coordinate, a color, and an intensity for each of one or more light sources.

13. The electronic device of claim 8 , wherein the AI light director is trained using an approximate loss that is based on comparing a raw image and a lighting direction applied to the raw image to an ideal relit image.

14. The electronic device of claim 8 , wherein the AI light director is trained using an actual loss that is based on comparing an AI-directed relit image generated using the lighting direction on a raw image in a multi-frame depth-based relighting engine to an ideal relit image.

15. A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:

capture a plurality of first images using a first image sensor;

capture a plurality of second images using a second image sensor;

estimate depth information based on at least one of the first images and at least one of the second images;

obtain information related to a lighting direction from an artificial intelligence (AI) light director, wherein the AI light director is trained to determine one or more lighting directions; and

generate at least one relit image using at least one of the first and second images based on the obtained information related to the lighting direction and the estimated depth information.

16. The machine-readable medium of claim 15 , wherein at least some of the first and second images are associated with different exposure conditions.

17. The machine-readable medium of claim 15 , further containing instructions that when executed cause the at least one processor to receive a user input that selects one relighting condition among a plurality of relighting conditions.

18. The machine-readable medium of claim 15 , wherein the information related to the lighting direction comprises an x-coordinate, a y-coordinate, a z-coordinate, a color, and an intensity for each of one or more light sources.

19. The machine-readable medium of claim 15 , wherein the AI light director is trained using an approximate loss that is based on comparing a raw image and a lighting direction applied to the raw image to an ideal relit image.

20. The machine-readable medium of claim 15 , wherein the AI light director is trained using an actual loss that is based on comparing an AI-directed relit image generated using the lighting direction on a raw image in a multi-frame depth-based relighting engine to an ideal relit image.