Apparatus and method for high dynamic range (HDR) image creation of dynamic scenes using graph cut-based labeling

1. A method comprising:

obtaining multiple image frames of a scene using at least one sensor of an electronic device, the multiple image frames including a first image frame, a second image frame having a longer exposure than the first image frame, and a third image frame having a shorter exposure than the second image frame, the second image frame captured in between the first and third image frames;

generating at least one label map that identifies pixels in the multiple image frames that are to be used in an image of the scene; and

generating the image of the scene using the pixels extracted from the image frames based on the at least one label maps;

wherein part of the first image frame corresponds to part of a saturated region of the second image frame and represents an occluded region that is blocked by at least one moving object in the first image frame; and

wherein generating the image of the scene comprises recovering image details in the part of the saturated region of the second image frame using a corresponding part of the third image frame.

2. The method of claim 1 , wherein:

the at least one label map comprises multiple labels; and

each label is associated with at least one corresponding pixel and comprises a discrete value that identifies one of the multiple image frames from which the at least one corresponding pixel is extracted.

3. The method of claim 1 , wherein:

generating the at least one label map comprises minimizing an overall cost function that is based on a data cost function and a smoothness cost function;

the data cost function jointly considers motion and well-exposedness; and

the smoothness cost function considers how each pixel's neighbors are labeled.

4. The method of claim 3 , wherein generating the at least one label map comprises performing a graph-cut optimization to identify labels in the at least one label map that minimize the overall cost function.

5. The method of claim 1 , wherein;

the at least one label map comprises first and second label maps; and

generating the image of the scene comprises:

combining the first and third image frames to produce a composite image frame using the first label map; and

combining the second image frame and the composite image frame using the second label map.

6. The method of claim 1 , wherein;

the at least one label map comprises a three-way label map; and

generating the image of the scene comprises combining the first, second, and third image frames using the three-way label map.

7. The method of claim 1 , wherein;

the at least one label map comprises first and second label maps; and

generating the image of the scene comprises:

combining the first and second image frames to produce a composite image frame using the first label map; and

combining the composite image frame and the third image frame using the second label map.

8. The method of claim 1 , further comprising:

aligning multiple captured image frames to produce at least first and second aligned image frames;

processing the first aligned image frame using histogram matching to generate a histogram-matched first aligned image frame; and

processing the first aligned image frame using tone mapping to generate a tone-matched first aligned image frame;

wherein generating the at least one label map comprises generating the at least one label map using the histogram-matched first aligned image frame and the second aligned image frame; and

wherein generating the image of the scene comprises using the tone-matched first aligned image frame, the second aligned image frame, and the at least one label map.

9. The method of claim 1 , wherein the first and third image frames have a common exposure.

10. The method of claim 1 , wherein one of:

the first image frame is captured before the second image and the third image frame is captured after the second image; or

the first image frame is captured after the second image and the third image frame is captured before the second image.

11. The method of claim 1 , wherein:

the multiple image frames include at least one additional image frame having a shorter exposure than the second image frame;

the at least one label map comprises multiple label maps;

a first of the label maps is generating using two of the image frames; and

each additional label map is generated using a composite image frame and another one of the image frames.

12. An electronic device comprising:

at least one sensor; and

at least one processing device configured to:

obtain multiple image frames of a scene using the at least one sensor, the multiple image frames including a first image frame, a second image frame having a longer exposure than the first image frame, and a third image frame having a shorter exposure than the second image frame, the second image frame captured in between the first and third image frames;

generate at least one label map that identifies pixels in the multiple image frames that are to be used in an image of the scene; and

generate the image of the scene using the pixels extracted from the image frames based on the at least one label map;

wherein part of the first image frame corresponds to part of a saturated region of the second image frame and represents an occluded region that is blocked by at least one moving object in the first image frame; and

wherein, to generate the image of the scene, the at least one processing device is configured to recover image details in the part of the saturated region of the second image frame using a corresponding part of the third image frame.

13. The electronic device of claim 12 , wherein the at least one sensor comprises at least one camera.

14. The apparatus of claim 12 , wherein:

to generate the at least one label map, the at least one processing device is configured to minimize an overall cost function that is based on a data cost function and a smoothness cost function;

the data cost function jointly considers motion and well-exposedness; and

the smoothness cost function considers how each pixel's neighbors are labeled.

15. The apparatus of claim 14 , wherein, to generate the at least one label map, the at least one processing device is configured to perform a graph-cut optimization to identify labels in the at least one label map that minimize the overall cost function.

16. The apparatus of claim 12 , wherein:

the at least one label map comprises first and second label maps; and

to generate the image of the scene, the at least one processing device is configured to:

combine the first and third image frames to produce a composite image frame using the first label map; and

combine the second image frame and the composite image frame using the second label map.

17. The apparatus of claim 12 , wherein:

the at least one label map comprises a three-way label map; and

to generate the image of the scene, the at least one processing device is configured to combine the first, second, and third image frames using the three-way label map.

18. The apparatus of claim 12 , wherein;

the at least one label map comprises first and second label maps; and

to generate the image of the scene, the at least one processing device is configured to:

combine the first and second image frames to produce a composite image frame using the first label map; and

combine the composite image frame and the third image frame using the second label map.

19. A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:

obtain multiple image frames of a scene using at least one sensor of the electronic device, the multiple image frames including a first image frame, a second image frame having a longer exposure than the first image frame, and a third image frame having a shorter exposure than the second image frame, the second image frame captured in between the first and third image frames;

generate at least one label map that identifies pixels in the multiple image frames that are to be used in an image of the scene; and

generate the image of the scene using the pixels extracted from the image frames based on the at least one label map;

wherein part of the first image frame corresponds to part of a saturated region of the second image frame and represents an occluded region that is blocked by at least one moving object in the first image frame; and

wherein the instructions that when executed cause the at least one processor to generate the image of the scene comprise instructions that when executed cause the at least one processor to recover image details in the part of the saturated region of the second image frame using a corresponding part of the third image frame.

20. The non-transitory machine-readable medium of claim 19 , wherein the instructions that when executed cause the at least one processor to generate the image of the scene comprise:

instructions that when executed cause the at least one processor to recover image details in another saturated region of the second image frame using a corresponding portion of the first image frame.

21. The non-transitory machine-readable medium of claim 19 , wherein:

the instructions that when executed cause the at least one processor to generate the at least one label map comprise instructions that when executed cause the at least one processor to minimize an overall cost function that is based on a data cost function and a smoothness cost function;

the data cost function jointly considers motion and well-exposedness; and

the smoothness cost function considers how each pixel's neighbors are labeled.

22. The non-transitory machine-readable medium of claim 21 , wherein the instructions that when executed cause the at least one processor to generate the at least one label map comprise:

instructions that when executed cause the at least one processor to perform a graph-cut optimization to identify labels in the at least one label map that minimize the overall cost function.

23. The non-transitory machine-readable medium of claim 19 , wherein:

the at least one label map comprises first and second label maps; and

the instructions that when executed cause the at least one processor to generate the image of the scene comprise instructions that when executed cause the at least one processor to:

combine the first and third image frames to produce a composite image frame using the first label map; and

combine the second image frame and the composite image frame using the second label map.

24. The non-transitory machine-readable medium of claim 19 , wherein:

the at least one label map comprises a three-way label map; and

the instructions that when executed cause the at least one processor to generate the image of the scene comprise instructions that when executed cause the at least one processor to combine the first, second, and third image frames using the three-way label map.

25. The non-transitory machine-readable medium of claim 19 , wherein:

the at least one label map comprises first and second label maps; and

the instructions that when executed cause the at least one processor to generate the image of the scene comprise instructions that when executed cause the at least one processor to:

combine the first and second image frames to produce a composite image frame using the first label map; and

combine the composite image frame and the third image frame using the second label map.