System and method for multi-frame temporal de-noising using image alignment
Methods, devices, and computer program products for multi-frame termporal de-noising using image alignment are describe. In one aspect, a method of capturing an image using a multi-frame temporal de-noising is described. The method includes capturing a plurality of frames and aligning the captured plurality of frames with each other. The method further includes determining a subset of frames of the captured plurality of frames, the subset determined based upon a focus quality of each frame of the plurality of frames. Finally, the method includes combining the subset of frames into a single image using a motion filter to reduce blurriness and ghosting.
1. An electronic device for capturing an image, the device comprising:
a memory component configured to store frames of image data, each frame having a plurality of pixels;
an electronic image sensor coupled to the memory component, the image sensor configured to capture frames of image data comprising a plurality of pixels;
a processor in data communication with the memory component, the processor configured to:
capture a plurality of frames of image data using the image sensor and store the plurality of frames in the memory component;
align the plurality of frames with each other by adding a first frame of the stored plurality of frames to a stack and calculating a fit between a first center frame at a center position of the stack and a frame adjacent to the first center frame;
determine a focus quality of each frame of the plurality of frames during the aligning the plurality of frames;
determine a subset of frames of the plurality of frames, the subset of frames determined based upon the determined focus quality of each frame of the plurality of frames; and
combine the subset of frames into a single image using a motion filter to reduce blurriness and ghosting.
2. The device of claim 1 , wherein the processor is configured to align the plurality of frames with each other using an affine fit process.
3. The device of claim 2 , wherein the affine fit process uses a six-parameter affine fit.
4. The device of claim 2 , wherein the processor is further configured to determine:
the focus quality of each frame of the plurality of frames during the aligning the plurality of frames using the affine fit process; and
a subset of frames of the plurality of frames by selecting a subset of frames of the captured plurality of frames based upon the determined focus quality of each of the captured plurality of frames.
5. The device of claim 4 , wherein the affine fit process is configured to use three sequential frames of the plurality of frames.
6. The device of claim 1 , wherein the processor is configured to determine a subset of frames of the captured plurality of frames by selecting a specified number of frames corresponding to the specified number of frames with the highest focus quality.
7. The device of claim 1 , wherein the motion filter is configured to replace a blurred object in a captured frame with a sharper version of the object from a reference frame.
8. The device of claim 1 , wherein the motion filter is configured to detect blurred portions of a captured frame and not use the blurred portions' to combine the subset of the plurality of frames into a single image.
9. A method of capturing an image, the method comprising:
capturing a plurality of frames of image data using an image sensor and storing the plurality of frames in a memory;
processing the captured plurality of frames to align the plurality of frames with each other by adding a first frame of the stored plurality of frames to a stack and calculating a fit between a center frame at a center position of the stack and a frame adjacent to the center frame;
determining a focus quality of each frame of the plurality of frames during the aligning the plurality of frames;
determining a subset of frames of the plurality of frames, the subset determined based upon the determined focus quality of each frame of the plurality of frames; and
combining the subset of frames into a single image using a motion filter to reduce blurriness and ghosting.
10. The method of claim 9 , wherein aligning the plurality of frames with each other comprises using an affine fit process.
11. The method of claim 10 , wherein the affine fit process uses a six-parameter affine fit.
12. The method of claim 10 , wherein determining the focus quality of each of the captured plurality of frames during the aligning the plurality of frames comprises using the affine fit process, and wherein determining the subset of frames of the plurality of frames comprises selecting a subset of frames of the captured plurality of frames based upon the determined focus quality of each of the captured plurality of frames.
13. The method of claim 12 , wherein the affine fit process is configured to use three sequential frames of the plurality of frames.
14. The method of claim 9 , wherein determining a subset of frames comprises selecting a specified number of frames corresponding to the specified number of frames with the highest focus quality.
15. The method of claim 9 , wherein the motion filter is configured to replace a blurred object in a captured frame with a sharper version of the object from a reference frame.
16. The method of claim 9 , wherein the motion filter is configured to detect blurred portions of a captured frame and not use the blurred portions to combine the subset of the plurality of frames into a single image.
17. An electronic device for capturing an image, the device comprising:
means for capturing a plurality of frames;
means for storing the captured plurality of frames;
means for processing the captured plurality of frames to align the plurality of frames with each other by adding a first frame of the stored plurality of frames to a stack and calculating a fit between a center frame at a center position of the stack and a frame adjacent to the center frame;
means for determining a focus quality of each frame of the plurality of frames during the aligning the plurality of frames;
means for determining a subset of frames of the plurality of frames, the subset determined based upon the determined focus quality of each frame of the plurality of frames; and
means for combining the subset of frames into a single image using a motion filter to reduce blurriness and ghosting.
18. The device of claim 17 , wherein means for processing the captured plurality of frames to align the plurality of frames with each other comprises means for using an affine fit process to align each of the captured frames with each other.
19. The device of claim 18 , wherein the affine fit process uses a six-parameter affine fit.
20. The device of claim 18 , wherein
the means for determining the focus quality of each frame of the plurality of frames during the aligning the plurality of frames comprises means for using the affine fit process, and wherein
the means for determining a subset of frames of the plurality of frames comprises means for selecting a subset of frames of the captured plurality of frames based upon the determined focus quality of each of the captured plurality of frames.
21. The device of claim 20 , wherein the affine fit process is configured to use three sequential frames of the plurality of frames.
22. The device of claim 17 , wherein means for determining a subset of frames comprises means for selecting a specified number of frames corresponding to the specified number of frames with the highest focus quality.
23. The device of claim 17 , wherein the motion filter is configured to replace a blurred object in a captured frame with a sharper version of the object from a reference frame.
24. The device of claim 17 , wherein the motion filter is configured to detect blurred portions of a captured frame and not use the blurred portions to combine the subset of the plurality of frames into a single image.
25. A non-transitory computer storage that stores executable program instructions that direct a wireless communications device to perform a process that comprises:
capturing a plurality of frames of image data using an image sensor and storing the plurality of frames in a memory;
processing the captured plurality of frames to align the plurality of frames with each other by adding a first frame of the stored plurality of frames to a stack and calculating a fit between a center frame at a center position of the stack and a frame adjacent to the center frame;
determining a focus quality of each frame of the plurality of frames based-upon during the aligning the plurality of frames;
determining a subset of frames of the plurality of frames, the subset determined based upon the determined focus quality of each frame of the plurality of frames; and
combining the subset of frames into a single image using a motion filter to reduce blurriness and ghosting.
26. The non-transitory computer storage of claim 25 , wherein aligning the plurality of frames with each other comprises using an affine fit process.
27. The non-transitory computer storage of claim 26 , wherein the affine fit process uses a six-parameter affine fit.
28. The non-transitory computer storage of claim 26 , wherein determining the focus quality of each frame of the plurality of frames is based upon during aligning the plurality of frames using the affine fit process, and wherein determining a subset of frames of the plurality of frames comprises selecting a subset of frames of the captured plurality of frames based upon the determined focus quality of each of the captured plurality of frames.
29. The non-transitory computer storage of claim 25 , wherein determining a subset of frames comprises selecting a specified number of frames corresponding to the specified number of frames with the highest focus quality.
30. The device of claim 1 , wherein the aligning the plurality of frames further comprises (i) updating the stack via adding a second frame of the stored plurality of frames to the stack and (ii) calculating a fit between a second center frame at the center position of the updated stack and a frame adjacent to the second center frame.