Dynamic reconstruction of high-resolution video from color-filtered low-resolution video-to-video super-resolution
View Patent ↗A method is provided of solving the dynamic super-resolution (SR) problem of reconstructing a high-quality set of monochromatic or color superresolved images from low-quality monochromatic, color, or mosaiced frames. The invention includes a joint method for simultaneous SR, deblurring, and demosaicing, this way taking into account practical color measurements encountered in video sequences. For the case of translational motion and common space-invariant blur, the proposed invention is based on a very fast and memory efficient approximation of the Kalman filter (KF). Experimental results on both simulated and real data are supplied, demonstrating the invention algorithms, and their strength.
1. A method of video-to-video dynamic super-resolution for reducing memory requirements and enhancing processing speed, color and resolution for grayscale and color sequences comprising:
a. using super-resolution, deblurring, and demosaicing simultaneously to incorporate color measurements encountered in video sequences;
b. using a pixel-wise mean-covariance approximate of a Kalman filter for an optimal pilot fused image on a high resolution grid;
c. applying advanced priors to both spatial and color-wise relationships for high quality recovery; and
d. using a non-causal process mode, wherein each said pilot fused image is derived as an optimal estimate incorporating information from all frames in a sequence using smoothed said approximated pixelwise Kalman filter to obtain an efficient algorithm.
2. The method of claim 1 , wherein a diagonal covariant matrix is applied to said approximated Kalman filter for enhanced processing speed.
3. The method of claim 1 , wherein said data fusion comprises:
a. causally estimating a state vector at time (t≧1) from a low-resolution frame at time (t≧1);
b. choosing a high-resolution frame at time (t=0) and said covariance matrix at time (t=0)=∈ 2 I, whereby ensuring said covariance matrix at time (t) is strictly positive definite at all times;
c. obtaining a low-resolution input image by:
i. setting t=t+1;
ii. applying a motion compensation by computing a forward-shifted matrix at time (t) and a forward shifted covariance matrix at time (t);
iii. updating said covariance matrix at time (t);
iv. updating said state vector at time (t); and
d. repeating step c.
4. The method of claim 1 , wherein said image interpolation, deblurring, and demosaicing comprise:
a. using a data fidelity penalty term, wherein said data fidelity penalty term is an L2 norm penalty term to enforce similarities between said pilot fused image and a high-resolution image estimate;
b. using a spatial luminance penalty term, wherein said spatial luminance penalty term is a penalty term to encourage sharp edges in a luminance component to said high-resolution image;
c. using a spatial chrominance penalty term, wherein said spatial chrominance penalty term is a term to encourage smoothness in a chrominance component of said high-resolution image; and
d. using an inter-color dependencies penalty term, wherein said inter-color dependencies penalty term is a term to encourage homogeneity of an edge location and orientation in different color bands.
5. The method according to claim 1 , wherein said lower-resolution color images comprise color filtered images, compressed color images, compressed color filtered images, and an image sequence with color artifacts.