Multi-frame depth-based multi-camera relighting of images

1. A method comprising:

capturing a plurality of first images using a first image sensor and a plurality of second images using a second image sensor;

generating a short single view, a short depth map, a long single view, and a long depth map from the plurality of first images and the plurality of second images;

generating a multi-frame depth map using the short single view, the short depth map, the long single view, and the long depth map;

generating a gain map using the multi-frame depth map and lighting conditions, wherein the lighting conditions introduce a relighting effect into the gain map;

generating a relit short single view and a relit long single view by applying the gain map to the short single view and the long single view; and

generating a relit image by fusing the relit short single view and the relit long single view.

2. The method of claim 1 , further comprising:

registering the plurality of first images;

fusing images of the plurality of first images having common exposure values into a first short frame and a first long frame;

registering the plurality of second images;

fusing images of the plurality of second images having common exposure values into a second short frame and a second long frame;

producing the short single view from the first short frame and the second short frame and producing the long single view from the first long frame and the second long frame; and

calculating the short depth map using the first short frame and the second short frame and calculating the long depth map using the first long frame and the second long frame.

3. The method of claim 1 , further comprising:

generating a motion map using a deghosting operation on the short single view and the long single view;

generating a saturation map using an exposure analysis on the short single view and the long single view;

generating a blend map by combining the motion map and the saturation map; and

generating the multi-frame depth map by applying the blend map to each of the short depth map and the long depth map.

4. The method of claim 1 , further comprising:

generating a normal map using the multi-frame depth map and the lighting conditions, wherein the normal map, for each location in a scene, contains a value of a surface normal at the location;

generating a light vector map from the multi-frame depth map and the lighting conditions, wherein the light vector map, for each location in the scene, indicates a light vector from a light source to the location; and

generating the gain map using a shading model to combine the normal map and the light vector map.

5. The method of claim 1 , further comprising:

generating a saturation map using an exposure analysis on the relit short single view and the relit long single view;

generating a blend map by combining the saturation map and a motion map; and

generating the relit image using a blend and tone map operation by applying the blend map to the relit short single view and the relit long single view.

6. The method of claim 1 , wherein:

the lighting conditions include a point source relighting, and

the relit image has light properties applied from a single light source.

7. The method of claim 1 , wherein:

the lighting conditions include a parallel light source effect, and the relit image has light properties applied from a plurality of parallel light sources.

8. The method of claim 7 , wherein the parallel light source effect is applied to create a foreground brightening effect.

9. The method of claim 7 , wherein the parallel light source effect is applied to create a background brightening effect.

10. An electronic device comprising:

a first image sensor configured to capture a plurality of first images;

a second image sensor configured to capture a plurality of second images; and

at least one processing device operatively coupled to the first image sensor and the second image sensor, the at least one processing device configured to:

generate a short single view, a short depth map, a long single view, and a long depth map from the plurality of first images and the plurality of second images;

generate a multi-frame depth map using the short single view, the short depth map, the long single view, and the long depth map;

generate a gain map using the multi-frame depth map and lighting conditions, wherein the lighting conditions introduce a relighting effect into the gain map;

apply the gain map to the short single view and the long single view in order to generate a relit short single view and a relit long single view; and

fuse the relit short single view and the relit long single view in order to generate a relit image.

11. The electronic device of claim 10 , wherein the at least one processing device is further configured to:

register the plurality of first images;

fuse images of the plurality of first images having common exposure values into a first short frame and a first long frame;

register the plurality of second images;

fuse images of the plurality of second images having common exposure values into a second short frame and a second long frame;

produce the short single view from the first short frame and the second short frame and produce the long single view from the first long frame and the second long frame; and

calculate the short depth map using the first short frame and the second short frame and calculate the long depth map using the first long frame and the second long frame.

12. The electronic device of claim 10 , wherein the at least one processing device is further configured to:

generate a motion map using a deghosting operation on the short single view and the long single view;

generate a saturation map using an exposure analysis on the short single view and the long single view;

combine the motion map and the saturation map in order to generate a blend map; and

apply the blend map to each of the short depth map and the long depth map in order to generate the multi-frame depth map.

13. The electronic device of claim 10 , wherein the at least one processing device is further configured to:

generate a normal map using the multi-frame depth map and the lighting conditions, wherein the normal map, for each location in a scene, contains a value of a surface normal at the location;

generate a light vector map from the multi-frame depth map and the lighting conditions, wherein the light vector map, for each location in the scene, indicates a light vector from a light source to the location; and

generate the gain map using a shading model to combine the normal map and the light vector map.

14. The electronic device of claim 10 , wherein the at least one processing device is further configured to:

generate a saturation map using an exposure analysis on the relit short single view and the relit long single view;

combine the saturation map and a motion map in order to generate a blend map; and

apply the blend map to the relit short single view and the relit long single view using a blend and tone map operation in order to generate the relit image.

15. The electronic device of claim 10 , wherein:

the lighting conditions include a point source relighting, and

the relit image has light properties applied from a single light source.

16. The electronic device of claim 10 , wherein:

the lighting conditions include a parallel light source effect, and

the relit image has light properties applied from a plurality of parallel light sources.

17. The electronic device of claim 16 , wherein the at least one processing device is configured to apply the parallel light source effect to create a foreground brightening effect.

18. The electronic device of claim 16 , wherein the at least one processing device is configured to apply the parallel light source effect to create a background brightening effect.

19. A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:

capture a plurality of first images using a first image sensor and a plurality of second images using a second image sensor;

generate a short single view, a short depth map, a long single view, and a long depth map from the plurality of first images and the plurality of second images;

generate a multi-frame depth map using the short single view, the short depth map, the long single view, and the long depth map;

generate a gain map using the multi-frame depth map and lighting conditions, wherein the lighting conditions introduce a relighting effect into the gain map;

apply the gain map to the short single view and the long single view in order to generate a relit short single view and a relit long single view; and

fuse the relit short single view and the relit long single view in order to generate a relit image.

20. The machine-readable medium of claim 19 , further containing instructions that when executed further cause the at least one processor to:

generate a normal map using the multi-frame depth map and the lighting conditions, wherein the normal map, for each location in a scene, contains a value of a surface normal at the location;

generate a light vector map from the multi-frame depth map and the lighting conditions, wherein the light vector map, for each location in the scene, indicates a light vector from a light source to the location; and

generate the gain map using a shading model to combine the normal map and the light vector map.