System for and method of sharpness enhancement using coding information and local spatial features
View Patent ↗In accordance with the present invention a method of and a system for enhancing image quality of a coded digital video signal representative of at least one frame in a digital video system are provided. The method comprises the steps of creating a usefulness metric identifying a limit to sharpness enhancement to be applied to the coded digital video signal, defining local spatial features in the frame, and identifying a frame type for the frame. The usefulness metric created in the creating step is: UME = 1 - M * ( 1 + q_scale N ) 2 * q_scale num_bits max ( q_scale num_bits ) wherein UME is the usefulness metric, q_scale is a quantization scale for a macroblock, num_bits is a number of bits to encode a luminance block, max is a function representing a maximum value for the frame, and M and N are scaling factors. The method further includes the steps of calculating a coding gain of each pixel in the frame based on the local spatial features and the usefulness metric in accordance with the frame type. Finally, the method also includes the steps of applying the coding gain to at least one sharpness enhancement algorithm, and generating an enhanced digital video signal by application of the sharpness enhancement algorithm.
1. A method of enhancing image quality of a coded digital video signal representative of at least one frame in a digital video system, said method comprising the steps of:
creating a usefulness metric identifying a limit to sharpness enhancement to be applied to said coded digital video signal;
defining local spatial features in a frame;
identifying a frame type for the frame;
calculating a coding gain of each pixel in the frame based on said local spatial features and said usefulness metric in accordance with the frame type;
applying said coding gain to at least one sharpness enhancement algorithm; and
generating an enhanced digital video signal by application of said sharpness enhancement algorithm.
2. The method of claim 1 , wherein the algorithm applied during the applying step is:
out_pixel=input_pixel+mpeg_gain*convolution_result,
wherein output_pixel is a final luminance value to be used for the pixel, input_pixel is a luminance value of the input pixel, mpeg_gain is the coding gain, and convolution_result is a high-pass filter output.
3. The method of claim 2 , wherein the high-pass filter output is obtained from a high-pass filter defined by
k
*
[
0
-
1
0
-
1
4
-
1
0
-
1
0
]
,
wherein k is a scaling factor in the range between 0 and 1.
4. The method of claim 1 , wherein said usefulness metric created in the creating step is:
UME
=
1
-
M
*
(
1
+
q_scale
N
)
2
*
q_scale
num_bits
max
(
q_scale
num_bits
)
wherein UME is the usefulness metric, q_scale is a quantization scale for a macroblock, num_bits is a number of bits to encode a luminance block, max is a function representing a maximum value for the frame, and M and N are scaling factors.
5. The method of claim 1 , wherein the local spatial feature defined by the defining step is a variance of pixel luminance values over an n×n window covering n×n pixels, said variance defined according to the equation:
var
(
i
,
j
)
=
∑
k
=
-
q
q
∑
m
=
-
q
q
pix
(
i
+
k
,
j
+
m
)
-
mean
wherein q=(n−1)/2, pix(i+k, j+m) is a pixel value at location (i+k, j+m) and mean is an average pixel value over said n×n window.
6. The method of claim 1 , wherein the frame type identified by the identifying step is an I-frame, and further wherein the step of calculating the coding gain includes the steps of:
calculating coding gain as:
UME
=
1
-
M
*
(
1
+
q_scale
N
)
2
*
q_scale
num_bits
max
(
q_scale
num_bits
)
wherein UME is the usefulness metric, q_scale is a quantization scale for a macroblock, num_bits is a number of bits to encode a luminance block, max is a function representing a maximum value for the frame, and M and N are scaling factors, if the local spatial feature is greater than a predetermined variance threshold and if the number of bits to encode the luminance block is not zero; and
calculating coding gain as zero if the local spatial feature is not greater than a predetermined variance threshold or if the number of bits to encode the luminance block is zero.
7. The method of claim 1 , wherein the frame type identified by the identifying step is a P-frame or a B-frame having skipped and non-skipped blocks using a reference frame and further wherein the step of calculating the coding gain includes the steps of:
retrieving coding gain from the reference frame based on a motion vector for a skipped block;
calculating coding gain as zero for a non-skipped block if the local spatial feature is equal to or less than a predetermined variance threshold; and
retrieving coding gain from the reference frame based on a motion vector for a non-skipped block if the local spatial feature is greater than a predetermined variance threshold.
8. The method of claim 1 , wherein the frame type identified by the identifying step is a B-frame having a reference block with a coding gain map and further wherein the step of calculating the coding gain includes the step of:
copying the coding gain map of the reference frame as the coding gain for the B-frame.
9. The method of claim 1 , wherein the applying step includes applying a peaking algorithm as the sharpness enhancement algorithm.
10. The method of claim 1 , wherein the applying step includes applying a spatial-domain algorithm as the sharpness enhancement algorithm.
11. A system for enhancing sharpness of a coded digital video signal representative of at least one frame, said system comprising:
a selector to select and extract statistical information from a coded digital video signal;
a usefulness metric generator to create a usefulness metric for said coded digital video signal after decoding, said usefulness metric identifies a limit to sharpness enhancement to be applied to a decoded video signal;
means for defining local spatial features in the frame;
means for identifying a frame type for the frame;
means for calculating a coding gain of each pixel in the frame based on said local spatial features and said usefulness metric in accordance with the frame type; and
a sharpness enhancer which applies a sharpness enhancement algorithm to the decoded digital video signal to improve sharpness of the signal based on said coding gain.
12. The system of claim 11 , wherein said sharpness enhancement algorithm is a peaking algorithm.
13. The system of claim 11 , wherein said sharpness enhancement algorithm is a spatial-domain algorithm.
14. The system of claim 11 , wherein the sharpness enhancement algorithm applied is:
out_pixel=input_pixel+mpeg_gain*convolution_result,
wherein output_pixel is a final luminance value to be used for the pixel, input_pixel is a luminance value of the input pixel, mpeg_gain is the coding gain, and convolution_result is a high-pass filter output.
15. The system of claim 11 , wherein the usefulness metric created is:
UME
=
1
-
M
*
(
1
+
q_scale
N
)
2
*
q_scale
num_bits
max
(
q_scale
num_bits
)
wherein UME is the usefulness metric, q_scale is a quantization scale for a macroblock, num_bits is a number of bits to encode a luminance block, max is a function representing a maximum value for the frame, and M and N are scaling factors.
16. The system of claim 11 , wherein the local spatial feature is defined as a variance of pixel luminance values over an n×n window covering n×n pixels, said variance defined according to the equation:
var
(
i
,
j
)
=
∑
k
=
-
q
q
∑
m
=
-
q
q
pix
(
i
+
k
,
j
+
m
)
-
mean
wherein q=(n−1)/2, pix(i+k, j+m) is a pixel value at location (i+k, j+m) and mean is an average pixel value over said n×n window.
17. The system of claim 11 , wherein when the frame type identified by the identifying means is an I-frame, the coding gain is calculated as:
UME
=
1
-
M
*
(
1
+
q_scale
N
)
2
*
q_scale
num_bits
max
(
q_scale
num_bits
)
wherein UME is the usefulness metric, q_scale is a quantization scale for a macroblock, num_bits is a number of bits to encode a luminance block, max is a function representing a maximum value for the frame, and M and N are scaling factors, if the local spatial feature is greater than a predetermined variance threshold and if the number of bits to encode the luminance block is not zero; and
coding gain is calculated as zero if the local spatial feature is not greater than a predetermined variance threshold or if the number of bits to encode the luminance block is zero.
18. The system of claim 11 , wherein when the frame type identified by the identifying means is a P-frame or a B-frame having skipped and non-skipped blocks using a reference frame, the coding gain is calculated by:
retrieving coding gain from the reference frame based on a motion vector for a skipped block;
calculating coding gain as zero for a non-skipped block if local spatial feature is equal to or less than a predetermined variance threshold; and
retrieving coding gain from the reference frame based on a motion vector for a non-skipped block if local spatial feature is greater than a predetermined variance threshold.
19. The system of claim 11 , wherein when the frame type identified by the identifying means is a B-frame having a reference block with a coding gain map, the coding gain is calculated by copying the coding gain map of the reference frame as the coding gain for the B-frame.