Methods for non-reference video-quality prediction
View Patent ↗A system for non-reference video-quality prediction includes a video-processing block to receive an input bitstream and to generate a first vector, and a neural network to provide a predicted-quality vector after being trained using training data. The training data includes the first vector and a second vector, and elements of the first vector include high-level features extracted from a high-level syntax processing of the input bitstream.
1. A system, comprising:
a video decoder configured to receive and decode an input bitstream to reconstruct a picture from the input bitstream, and to generate a first vector comprising features extracted from the input bitstream by the video decoder and features determined by the video decoder during reconstruction of the picture; and
a neural network configured to generate a second vector, based on the first vector, comprising one or more metrics representing a predicted quality of the picture reconstructed by the video decoder,
wherein one or more parameters of the neural network are trained by:
computing a first training vector based on a test bitstream that has been decoded,
generating, using the video decoder, a second training vector comprising features extracted from the test bitstream and determined by the video decoder,
generating, using the neural network, a third training vector based on the second training vector,
computing a prediction loss between the first training vector and the third training vector, and
updating the one or more parameters based on the prediction loss.
2. The system of claim 1 , wherein the video decoder is configured to extract the features from the input bitstream by parsing syntax elements in the input bitstream.
3. The system of claim 2 , wherein the syntax elements comprise at least one of sequence parameter sets, picture parameter sets, video parameter sets, picture headers, slice headers, adaptation parameter sets, or supplemental enhancement information messages.
4. The system of claim 2 , wherein the features extracted from the input bitstream comprise at least one of a transcode indicator, a codec type, a picture coding type, a picture resolution, a frame rate, a bit depth, a chroma format, a compressed picture size, a high-level quantization parameter, an average temporal distance, or a temporal layer identifier.
5. The system of claim 1 , wherein the features determined by the video decoder during reconstruction of the picture comprise at least one of a percentage of intra-coded blocks in the picture, a percentage of inter-coded blocks in the picture, an average block-level quantization parameter for the picture, a maximum block-level quantization parameter for the picture, a minimum block-level quantization for the picture, a standard deviation of horizontal-motion vectors for the picture, an average motion-vector size for the picture, an average absolute amplitude of low-frequency inverse quantized transform coefficients for the picture, an average absolute amplitude of high-frequency inverse quantized-transform coefficients for the picture, a standard deviation of a prediction residual for the picture, a root-mean-squared error value between reconstructed pictures before and after in-loop filters, a standard deviation of the reconstructed picture after in-loop filters, or an edge sharpness of the reconstructed picture after in-loop filters.
6. The system of claim 1 , wherein the one or more metrics representing the predicted quality of the reconstructed picture comprise at least one of a peak signal noise ratio, a structural similarity index measure, a multiscale similarity index measure, a video multimethod assessment fusion, or a mean opinion score.
7. The system of claim 6 , wherein an output layer of the neural network is configured to generate root-mean-squared-error values.
8. The system of claim 7 , wherein the root-mean-squared-error values are converted into peak signal noise ratio values.
9. A method, comprising:
receiving and decoding, by a video decoder, an input bitstream to reconstruct a picture from the input bitstream;
generating, by the video decoder, a first vector comprising features extracted from the input bitstream by the video decoder and determined during reconstruction of the picture by the video decoder;
generating, using a neural network, a second vector comprising one or more metrics representing a predicted quality of the reconstructed picture, wherein the second vector is based on the first vector,
wherein one or more parameters of the neural network are trained by:
computing a first training vector based on a test bitstream that has been decoded;
generating, using the video decoder, a second training vector comprising features extracted from the test bitstream and determined by the video decoder;
generating, using the neural network, a third training vector based on the second training vector;
computing a prediction loss between the first training vector and the third training vector; and
updating the one or more parameters based on the prediction loss.
10. The method of claim 9 , wherein the features are extracted from the input bitstream by parsing syntax elements in the input bitstream.
11. The method of claim 10 , wherein the syntax elements comprise at least one of sequence parameter sets, picture parameter sets, video parameter sets, picture headers, slice headers, adaptation parameter sets, or supplemental enhancement information messages.
12. The method of claim 10 , wherein the features extracted from the input bitstream comprise at least one of a transcode indicator, a codec type, a picture coding type, a picture resolution, a frame rate, a bit depth, a chroma format, a compressed picture size, a high-level quantization parameter, an average temporal distance, or a temporal layer identifier.
13. The method of claim 9 , wherein the features determined by the video decoder during reconstruction of the picture comprise at least one of a percentage of intra-coded blocks in the picture, a percentage of inter-coded blocks in the picture, an average block-level quantization parameter for the picture, a maximum block-level quantization parameter for the picture, a minimum block-level quantization for the picture, a standard deviation of horizontal-motion vectors for the picture, an average motion-vector size for the picture, an average absolute amplitude of low-frequency inverse quantized transform coefficients for the picture, an average absolute amplitude of high-frequency inverse quantized-transform coefficients for the picture, a standard deviation of a prediction residual for the picture, a root-mean-squared error value between reconstructed pictures before and after in-loop filters, a standard deviation of the reconstructed picture after in-loop filters, or an edge sharpness of the reconstructed picture after in-loop filters.
14. The method of claim 9 , wherein the one or more metrics representing the predicted quality of the reconstructed picture comprise at least one of a peak signal noise ratio, a structural similarity index measure, a multiscale similarity index measure, a video multimethod assessment fusion, or a mean opinion score.
15. The method of claim 14 , wherein an output layer of the neural network is configured to generate root-mean-squared-error values.
16. The method of claim 15 , wherein the root-mean-squared-error values are converted into peak signal noise ratio values.