Video processing system and method for automatic enhancement of digital video
View Patent ↗The automated video enhancement system and method includes a system and method for automatically enhancing video. The automated video enhancement method uses frame-to-frame motion estimation as the basis of the video enhancement. Motion estimation includes the computation of global motion (such as camera motion) and the computation of local motion (such as pixel motion). The automated video enhancement method includes generating global alignment transforms, generating optic flow vectors, and using these global alignment transforms and optic flow vectors to enhance the video. The invention also includes video processing and enhancement techniques that use the frame-to-frame motion estimation. These techniques include a deinterlace process, a denoise process, and a warp stabilization process that performs both damped and locked stabilization.
1. A method for processing a video containing a video frame sequence, comprising:
using a computing device having a processor to perform the following:
performing global motion computation to using an affine transformation to generate global alignment affine transforms; and
performing warp stabilization on the video frame sequence using the global alignment affine transforms.
2. The method as set forth in claim 1 , wherein performing warp stabilization further comprises:
discovering any blurry frames within the video frame sequence;
discarding the blurry frames;
synthesizing new frames using the global alignment affine transforms and
replacing the discarded blurry frames with the new frames.
3. The method as set forth in claim 1 , further comprising:
using the global alignment affine transforms to determine moving pixels in a video frame of the video frame sequence; and
discarding the moving pixels within the video frame.
4. The method as set forth in claim 3 , further comprising filling in the discarded moving pixels within the video frame with pixels borrowed from frames neighboring the video frame.
5. The method as set forth in claim 1 , wherein the warped stabilization is a damped stabilization, and performing the warp stabilization further comprises:
determining a curve corresponding to an actual motion of a camera that captured the video;
applying a smoothing operation to the actual camera motion to yield an intended camera motion; and
determining the difference between the intended camera motion and the actual camera motion to recover camera jitter.
6. The method as set forth in claim 1 , wherein the warped stabilization is a locked stabilization, and performing the warp stabilization further comprises:
determining a video frame from the video frame sequence to use as a reference frame; and
canceling out moving pixels in a video frame discovered by the global alignment affine transforms.
7. The method as set forth in claim 6 , wherein the reference frame is a first frame of the video frame sequence.
8. The method as set forth in claim 6 , further comprising filling in the canceled pixels by building a mosaic.
9. The method as set forth in claim 8 , wherein building a mosaic further comprises using similar pixels from nearby video frames to fill in the canceled pixels.