Techniques for combining image frames captured using different exposure settings into blended images

1. A method for multi-frame blending, the method comprising:

obtaining at least two image frames of a scene, one of the image frames associated with a shorter exposure time and a higher sensitivity and representing a reference image frame, at least one other of the image frames associated with a longer exposure time and a lower sensitivity and representing at least one non-reference image frame; and

blending the reference and non-reference image frames into a blended image such that (i) one or more motion regions of the blended image are based more on the reference image frame and (ii) one or more stationary regions of the blended image are based more on the at least one non-reference image frame.

2. The method of claim 1 , wherein a ratio between the longer and shorter exposure times substantially matches a ratio between the higher and lower sensitivities so that the image frames have a substantially equal overall brightness.

3. The method of claim 1 , further comprising:

selecting one of the at least one other of the image frames as the reference image frame in response to determining that the image frame associated with the shorter exposure time and the higher sensitivity is of insufficient quality.

4. The method of claim 1 , further comprising:

obtaining multiple image frames associated with the shorter exposure time and the higher sensitivity; and

selecting one of the multiple image frames associated with the shorter exposure time and the higher sensitivity as the reference image frame.

5. The method of claim 1 , wherein:

the image frames comprise the reference image frame and multiple non-reference image frames; and

the reference image frame is associated with a capture time that is positioned in between capture times associated with at least two of the non-reference image frames.

6. The method of claim 1 , further comprising:

generating the reference image frame by performing a nonlinear transformation of a reference input image frame based on at least one histogram associated with at least one non-reference input image frame.

7. The method of claim 6 , wherein performing the nonlinear transformation comprises:

generating different histogram maps based on the reference input image frame and different non-reference input image frames;

wherein the histogram maps are generated by excluding motion pixels from consideration and generating the histogram maps based on non-excluded pixels.

8. The method of claim 7 , wherein generating at least one of the histogram maps comprises:

identifying absolute differences between pixel values from two of the input image frames;

generating a histogram based on the absolute differences;

identifying one or more motion threshold values based on the histogram;

excluding any pixel from the two input image frames having at least one pixel value that exceeds at least one of the one or more motion threshold values; and

generating the at least one histogram map based on pixels from the two input image frames that are not excluded.

9. The method of claim 1 , wherein blending the reference and non-reference image frames comprises:

generating multiple motion maps using the reference and non-reference image frames, the motion maps comprising (i) one or more first motion maps each associated with differences between the reference image frame and one of the at least one non-reference image frame and (ii) a second motion map associated with the reference image frame.

10. The method of claim 9 , wherein:

the image frames comprise the reference image frame and multiple non-reference image frames;

the one or more first motion maps comprise multiple first motion maps; and

the method further comprises generating the second motion map by:

combining the first motion maps to generate a combined first motion map;

for pixels or regions where the combined first motion map indicates little or no motion is occurring, minimizing a contribution of the reference image frame to the blended image in the second motion map; and

for pixels or regions where the combined first motion map indicates motion is occurring, maximizing the contribution of the reference image frame to the blended image in the second motion map.

11. The method of claim 1 , further comprising:

applying contour reduction to the blended image to reduce contour artifacts in the blended image.

12. The method of claim 11 , wherein:

the image frames comprise the reference image frame and multiple non-reference image frames; and

applying contour reduction comprises:

identifying where motion maps associated with the non-reference image frames do or do not consistently identify motion; and

identifying pixels or regions of the blended image in which to apply contour reduction based on where the motion maps do not consistently identify motion.

13. The method of claim 12 , wherein applying contour reduction further comprises:

identifying image differences between at least luminance data associated with the reference image frame and with the blended image;

normalizing the image differences based on detected object edges in the reference image frame to produce normalized image differences;

filtering the normalized image differences to produce filtered normalized image differences; and

subtracting the filtered normalized image differences or scaled versions of the filtered normalized image differences from the blended image only for the identified pixels or regions of the blended image in which contour reduction is applied.

14. An electronic device comprising:

at least one image sensor; and

at least one processor configured to:

obtain at least two image frames of a scene using the at least one image sensor, one of the image frames associated with a shorter exposure time and a higher sensitivity and representing a reference image frame, at least one other of the image frames associated with a longer exposure time and a lower sensitivity and representing at least one non-reference image frame; and

blend the reference and non-reference image frames into a blended image such that (i) one or more motion regions of the blended image are based more on the reference image frame and (ii) one or more stationary regions of the blended image are based more on the at least one non-reference image frame.

15. The electronic device of claim 14 , wherein a ratio between the longer and shorter exposure times substantially matches a ratio between the higher and lower sensitivities so that the image frames have a substantially equal overall brightness.

16. The electronic device of claim 14 , wherein the at least one processor is further configured to select one of the at least one other of the image frames as the reference image frame in response to determining that the image frame associated with the shorter exposure time and the higher sensitivity is of insufficient quality.

17. The electronic device of claim 14 , wherein the at least one processor is further configured to:

obtain multiple image frames associated with the shorter exposure time and the higher sensitivity; and

select one of the multiple image frames associated with the shorter exposure time and the higher sensitivity as the reference image frame.

18. The electronic device of claim 14 , wherein:

the image frames comprise the reference image frame and multiple non-reference image frames; and

the reference image frame is associated with a capture time that is positioned in between capture times associated with at least two of the non-reference image frames.

19. The electronic device of claim 14 , wherein:

the at least one processor is further configured to generate the reference image frame; and

to generate the reference image frame, the at least one processor is configured to perform a nonlinear transformation of a reference input image frame based on at least one histogram associated with at least one non-reference input image frame.

20. The electronic device of claim 19 , wherein:

to perform the nonlinear transformation, the at least one processor is configured to generate different histogram maps based on the reference input image frame and different non-reference input image frames; and

to generate the histogram maps, the at least one processor is configured to exclude motion pixels from consideration and to generate the histogram maps based on non-excluded pixels.

21. The electronic device of claim 20 , wherein, to generate at least one of the histogram maps, the at least one processor is configured to:

identify absolute differences between pixel values from two of the input image frames;

generate a histogram based on the absolute differences;

identify one or more motion threshold values based on the histogram;

exclude any pixel from the two input image frames having at least one pixel value that exceeds at least one of the one or more motion threshold values; and

generate the at least one histogram map based on pixels from the two input image frames that are not excluded.

22. The electronic device of claim 14 , wherein:

to blend the reference and non-reference image frames, the at least one processor is configured to generate multiple motion maps using the reference and non-reference image frames; and

the motion maps comprise (i) one or more first motion maps each associated with differences between the reference image frame and one of the at least one non-reference image frame and (ii) a second motion map associated with the reference image frame.

23. The electronic device of claim 22 , wherein:

the image frames comprise the reference image frame and multiple non-reference image frames;

the one or more first motion maps comprise multiple first motion maps;

the at least one processor is further configured to generate the second motion map; and

to generate the second motion map, the at least one processor is configured to:

combine the first motion maps to generate a combined first motion map;

for pixels or regions where the combined first motion map indicates little or no motion is occurring, minimize a contribution of the reference image frame to the blended image in the second motion map; and

for pixels or regions where the combined first motion map indicates motion is occurring, maximize the contribution of the reference image frame to the blended image in the second motion map.

24. The electronic device of claim 14 , wherein the at least one processor is further configured to apply contour reduction to the blended image to reduce contour artifacts in the blended image.

25. The electronic device of claim 24 , wherein:

the image frames comprise the reference image frame and multiple non-reference image frames; and

to apply contour reduction, the at least one processor is configured to:

identify where motion maps associated with the non-reference image frames do or do not consistently identify motion; and

identify pixels or regions of the blended image in which to apply contour reduction based on where the motion maps do not consistently identify motion.

26. The electronic device of claim 25 , wherein, to apply contour reduction, the at least one processor is further configured to:

identify image differences between at least luminance data associated with the reference image frame and with the blended image;

normalize the image differences based on detected object edges in the reference image frame to produce normalized image differences;

filter the normalized image differences to produce filtered normalized image differences; and

subtract the filtered normalized image differences or scaled versions of the filtered normalized image differences from the blended image only for the identified pixels or regions of the blended image in which contour reduction is applied.

27. A method for contour reduction, the method comprising:

obtaining a blended image that is generated by combining a reference image frame and multiple non-reference image frames;

identifying where motion maps associated with the non-reference image frames do or do not consistently identify motion relative to the reference frame;

identifying pixels or regions of the blended image in which to apply contour reduction based on where the motion maps do not consistently identify motion; and

applying contour reduction to the identified pixels or regions to produce a processed blended image.

28. The method of claim 27 , wherein applying contour reduction comprises:

identifying image differences between at least luminance data associated with the reference image frame and with the blended image;

normalizing the image differences based on detected object edges in the reference image frame to produce normalized image differences;

filtering the normalized image differences to produce filtered normalized image differences; and

subtracting the filtered normalized image differences or scaled versions of the filtered normalized image differences from the blended image only for the identified pixels or regions of the blended image in which contour reduction is applied.

29. The method of claim 28 , wherein identifying the image differences comprises identifying the image differences between the luminance data associated with the reference image frame and with the blended image, without identifying image differences between chrominance data associated with the reference image frame and with the blended image.

30. The method of claim 28 , wherein filtering the normalized image differences comprises low-pass filtering the normalized image differences.

31. The method of claim 30 , further comprising:

performing edge detection filtering to identify the object edges in the reference image frame.

32. The method of claim 31 , wherein:

performing edge detection filtering comprises performing edge detection filtering using pixels in a moving 3×3 window; and

filtering the normalized image differences comprises filtering the normalized image differences using a 7x7 uniform filter.

33. The method of claim 27 , wherein identifying where the motion maps associated with the non-reference image frames do or do not consistently identify motion comprises, for each pixel or region:

determining a ratio based on contributions of the non-reference image frames to the blended image frame; and

determining whether the ratio indicates that contour artifacts are likely to form based on a contour reduction threshold.

34. The method of claim 27 , further comprising:

generating the blended image by combining the non-reference image frames and the reference image frame, the reference image frame associated with a shorter exposure time and a higher sensitivity, the non-reference image frames associated with a longer exposure time and a lower sensitivity.