Method and apparatus for spatio-temporal subband video enhancement with small time delay
View Patent ↗Video processing method and means for enhancing a video stream, by computing transform coefficients using a spatio-temporal transform comprising a spatial subband transform and a causal time wavelet transform performing filterings with multiscale causal wavelets, modifying the transform coefficients with a nonlinear processing, and computing a processed video stream from the modified transform coefficients using a spatio-temporal reconstruction transform comprising an inverse subband transform and a short delay inverse time wavelet transform, where the short delay inverse time wavelet transform is implemented with wavelet filters modified with window functions to control the processing delay of the entire video processing method.
1. A method for enhancing a digital video stream of digital images, comprising:
(a) transforming the digital images of the video stream into spatio-temporal subband images by processing circuitry of a video apparatus, the transforming comprising a causal time wavelet transform using filterings by multiscale causal wavelets and a spatial subband transform;
(b) applying causal non-linear operators to said spatio-temporal subband images by the processing circuitry, to compute modified spatio-temporal subband images; and
(c) applying to said modified spatio-temporal subband images a spatio-temporal reconstruction by the processing circuitry to compute enhanced video images delayed by a predetermined time delay of d images of the video stream, d being a non-negative integer, wherein said spatio-temporal reconstruction comprises:
i. an inverse of said spatial subband transform; and
ii. an inverse of said time wavelet transform using filterings with multiscale reconstruction wavelets each multiplied with either a window function of support [−d; 0], or with a sum of nonnegative translates of said window function.
2. A video enhancement method according to claim 1 wherein transforming the digital images into spatio-temporal subband images comprises:
(a) applying said causal time wavelet transform to said digital images to compute temporal subband images; and
(b) applying said spatial subband transform to said temporal subband images to compute said spatio-temporal subband images.
3. A video enhancement method according to claim 1 , wherein said spatio-temporal reconstruction comprises:
(a) applying said inverse time wavelet transform to said modified spatio-temporal subband images to compute spatial subband images; and
(b) applying an inverse of said spatial subband transform to said subband images to obtain the enhanced video images.
4. A video enhancement method according to claim 1 , wherein said spatio-temporal reconstruction transform comprises:
(a) applying said inverse time wavelet transform to said modified spatio-temporal subband images to compute spatial subband images;
(b) applying spatial non-linear operators to said spatial subband images to compute modified spatial subband images; and
(c) applying an inverse of said spatial subband transform to said modified subband images to obtain the enhanced video images.
5. A video enhancement method according to claim 1 , wherein said spatial subband transform comprises a wavelet transform.
6. A video enhancement method according to claim 1 , wherein said spatial subband transform comprises a bandlet transform.
7. A video enhancement method according to claim 1 , wherein said non-linear operators comprise thresholding operators.
8. A video enhancement method according to claim 1 , wherein d=0.
9. A video enhancement method according to claim 1 , wherein d>0.
10. A video enhancement method according to claim 1 , wherein said multiscale causal wavelets comprise boundary wavelets designed for a biorthogonal wavelet transform on [0,+∞), and wherein said reconstruction multiscale wavelets comprise reversed dual boundary wavelets.
11. A video enhancement method according to claim 1 , wherein said causal multiscale wavelets comprise Haar wavelets.
12. A video enhancement method according to claim 11 , wherein said causal time wavelet transform comprises:
(a) computing recursively for each scale 1 to J a multiscale average as an average of a current time multiscale average and a past multiscale average read from a buffer, or of a current time input frame and a past input frame read from a buffer;
(b) computing time multiscale wavelet coefficients as weighted differences of two of said current time multiscale averages or as a weighted difference of a current time input frame and a multiscale average.
13. A video apparatus, comprising: processing circuitry arranged for
(a) transforming digital images of a video stream into spatio-temporal subband images, the transforming comprising a causal time wavelet transform using filterings by multiscale causal wavelets and a spatial subband transform;
(b) applying causal non-linear operators to said spatio-temporal subband images to compute modified spatio-temporal subband images; and
(c) applying to said modified spatio-temporal subband images a spatio-temporal reconstruction to compute enhanced video images delayed by a predetermined time delay of d images of the video stream, d being a non-negative integer, wherein said spatio-temporal reconstruction comprises:
i. an inverse of said spatial subband transform; and
ii. an inverse of said time wavelet transform using filterings with multiscale reconstruction wavelets each multiplied with either a window function of support [−d; 0], or with a sum of nonnegative translates of said window function.
14. A video apparatus according to claim 13 , wherein transforming the digital images into spatio-temporal subband images comprises:
(a) applying said causal time wavelet transform to said digital images to compute temporal subband images; and
(b) applying said spatial subband transform to said temporal subband images to compute said spatio-temporal subband images.
15. A video apparatus according to claim 13 , wherein said spatio-temporal reconstruction comprises
(a) applying said inverse time wavelet transform to said modified spatio-temporal subband images to compute spatial subband images; and
(b) applying an inverse of said spatial subband transform to said subband images to obtain the enhanced video images.
16. A video apparatus according to claim 13 , wherein said spatio-temporal reconstruction transform comprises:
(a) applying said inverse time wavelet transform to said modified spatio-temporal subband images to compute spatial subband images;
(b) applying spatial non-linear operators to said spatial subband images to compute modified spatial subband images; and
(c) applying an inverse of said spatial subband transform to said modified subband images to obtain the enhanced video images.
17. A video apparatus according to claim 13 , wherein said causal multiscale wavelets comprise Haar wavelets.
18. A video apparatus according to claim 17 , wherein said causal time wavelet transform comprises:
(a) computing recursively for each scale 1 to J a multiscale average as an average of a current time multiscale average and a past multiscale average read from a buffer, or of a current time input frame and a past input frame read from a buffer;
(b) computing time multiscale wavelet coefficients as weighted differences of two of said current time multiscale averages or as a weighted difference of a current time input frame and a multiscale average.
19. A non-transitory machine-readable medium having stored therein a computer program product, wherein the program product comprises instructions to carry out the following when said program product is run in processing circuitry of a video apparatus:
(a) transforming digital images of a video stream into spatio-temporal subband images, the transforming comprising a causal time wavelet transform using filterings by multiscale causal wavelets and a spatial subband transform;
(b) applying causal non-linear operators to said spatio-temporal subband images to compute modified spatio-temporal subband images; and
(c) applying to said modified spatio-temporal subband images a spatio-temporal reconstruction to compute enhanced video images delayed by a predetermined time delay of d images of the video stream, d being a non-negative integer, wherein said spatio-temporal reconstruction comprises:
i. an inverse of said spatial subband transform; and
ii. an inverse of said time wavelet transform using filterings with multiscale reconstruction wavelets each multiplied with either a window function of support [−d; 0], or with a sum of nonnegative translates of said window function.