Determining adaptive quantization matrices using machine learning for video coding
Techniques related to adaptive quantization matrix selection using machine learning for video coding are discussed. Such techniques include applying a machine learning model to generate an estimated quantization parameter for a frame and selecting a set of quantization matrices for encode of the frame from a number of sets of quantization matrices based on the estimated quantization parameter.
1. A system comprising:
a memory to store at least a portion of a current frame of a video sequence; and
one or more processors coupled to the memory, the one or more processors to:
determine an estimated quantization parameter (QP) for the current frame using the current frame and one or more frames subsequent to the current frame;
determine the estimated QP is a within a particular sub-range of a plurality of sub-ranges of an available QP range;
selecting a quantization matrix for the current frame from a plurality of available quantization matrices based on the estimated QP being within the particular sub-range;
encode the current frame using the selected quantization matrix to generate at least a portion of a bitstream;
determine a second estimated QP, greater than or less than the estimated QP, for a second frame of the video sequence temporally prior to the current frame, the second estimated QP corresponding to a second sub-range of the plurality of sub-ranges and a second quantization matrix; and
encode the second frame using the selected quantization matrix in response to the estimated QP for the current frame being a quantization matrix switching QP.
2. The system of claim 1 , wherein the selected quantization matrix has a first entry corresponding to a DC coefficient and a second entry corresponding to an AC coefficient both having a same value.
3. The system of claim 1 , wherein the plurality of sub-ranges comprises at least a first sub-range extending from a lowest available QP of the available QP range to a first QP, the second sub-range extending from a second QP, greater than the first QP, to a third QP, greater than the second QP, and a third sub-range extending from a fourth QP, greater than the third QP, to a greatest available QP of the available QP range.
4. The system of claim 3 , wherein a first quantization matrix for the first sub-range is a flat matrix, a second quantization matrix for the second sub-range comprises a first difference between a first entry corresponding to a DC coefficient and a second entry corresponding to an AC coefficient, and a third quantization matrix for the third sub-range comprises a second difference between a third entry corresponding to the DC coefficient and a fourth entry corresponding to the AC coefficient, the second difference being greater than the first difference.
5. The system of claim 4 , wherein the first difference is zero.
6. The system of claim 1 , wherein the one or more processors are further to:
determine a frame QP for the current frame was generated based on rate control applied using a further quantization matrix; and
adjust the frame QP to a second frame QP based on a QP adjustment corresponding to the selected quantization matrix and the further quantization matrix, wherein the current frame is encoded using the adjusted frame QP.
7. The system of claim 6 , wherein the QP adjustment is negative in response to a first sub-range for the second quantization matrix having lower QPs than the particular sub-range or positive in response to the first sub-range having greater QPs than the particular sub-range.
8. The system of claim 1 , wherein the quantization matrix switching QP comprises a maximum QP value in the second sub-range.
9. The system of claim 1 , wherein the quantization matrix switching QP comprises a minimum QP value in the second sub-range.
10. The system of claim 1 , wherein determining the estimated quantization parameter for the current frame comprises:
generating a set of features for each of the current frame and the one or more frames subsequent to the current frame;
applying a machine learning model to each of the sets of features to generate a modeled QP for each of the current frame and the one or more frames subsequent to the current frame; and
combining the modeled QPs to determine the estimated QP for the current frame.
11. The system of claim 10 , wherein a first set of features of the sets of features comprises a number of bits generated by a look ahead encode of the current frame, a proportion of syntax bits of the number of bits from the look ahead encode, a proportion of intra coded blocks from the look ahead encode, and a prediction distortion from the look ahead encode.
12. The system of claim 1 , wherein the one or more processors are further to:
define the sub-ranges of the available QP range;
generate a set of initial quantization matrices for each of the sub-ranges;
modify DC and low AC frequency values of the sets of initial quantization matrices to generate a plurality of quantization matrices for testing comprising the sets of initial quantization matrices and the modified sets of initial quantization matrices;
encode test video sequences using the plurality of quantization matrices for testing; and
select a final set of quantization matrices for each of the sub-ranges based on perceptual quality scoring of said encodings.
13. A method comprising:
determining an estimated quantization parameter (QP) for a current frame of a video sequence using the current frame and one or more frames subsequent to the current frame, wherein determining the estimated quantization parameter for the current frame comprises:
generating a set of features for each of the current frame and the one or more frames subsequent to the current frame;
applying a machine learning model to each of the sets of features to generate a modeled QP for each of the current frame and the one or more frames subsequent to the current frame; and
combining the modeled QPs to determine the estimated QP for the current frame;
determining the estimated QP is a within a particular sub-range of a plurality of sub-ranges of an available QP range;
selecting a quantization matrix for the current frame from a plurality of available quantization matrices based on the estimated QP being within the particular sub-range; and
encoding the current frame using the selected quantization matrix to generate at least a portion of a bitstream.
14. The method of claim 13 , wherein the selected quantization matrix has a first entry corresponding to a DC coefficient and a second entry corresponding to an AC coefficient both having a same value.
15. The method of claim 13 , wherein the plurality of sub-ranges comprises at least a first sub-range extending from a lowest available QP of the available QP range to a first QP, a second sub-range extending from a second QP, greater than the first QP, to a third QP, greater than the second QP, and a third sub-range extending from a fourth QP, greater than the third QP, to a greatest available QP of the available QP range, wherein a first quantization matrix for the first sub-range is a flat matrix, a second quantization matrix for the second sub-range comprises a first difference between a first entry corresponding to a DC coefficient and a second entry corresponding to an AC coefficient, and a third quantization matrix for the third sub-range comprises a second difference between a third entry corresponding to the DC coefficient and a fourth entry corresponding to the AC coefficient, the second difference being greater than the first difference.
16. The method of claim 13 , further comprising:
determining a frame QP for the current frame was generated based on rate control applied using a further quantization matrix; and
adjusting the frame QP to a second frame QP based on a QP adjustment corresponding to the selected quantization matrix and the further quantization matrix, wherein the current frame is encoded using the adjusted frame QP.
17. The method of claim 13 , further comprising:
determining a second estimated QP, greater than or less than the estimated QP, for a second frame of the video sequence temporally prior to the current frame, the second estimated QP corresponding to a second sub-range of the plurality of sub-ranges and a second quantization matrix; and
encoding the second frame using the selected quantization matrix in response to the estimated QP for the current frame being a quantization matrix switching QP.
18. At least one non-transitory machine readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to perform video coding by:
determining an estimated quantization parameter (QP) for a current frame of a video sequence using the current frame and one or more frames subsequent to the current frame;
determining the estimated QP is a within a particular sub-range of a plurality of sub-ranges of an available QP range;
selecting a quantization matrix for the current frame from a plurality of available quantization matrices based on the estimated QP being within the particular sub-range; and
encoding the current frame using the selected quantization matrix to generate at least a portion of a bitstream;
defining the sub-ranges of the available QP range;
generating a set of initial quantization matrices for each of the sub-ranges;
modifying DC and low AC frequency values of the sets of initial quantization matrices to generate a plurality of quantization matrices for testing comprising the sets of initial quantization matrices and the modified sets of initial quantization matrices;
encoding test video sequences using the plurality of quantization matrices for testing; and
selecting a final set of quantization matrices for each of the sub-ranges based on perceptual quality scoring of said encodings.
19. The at least one non-transitory machine readable medium of claim 18 , wherein the plurality of sub-ranges comprises at least a first sub-range extending from a lowest available QP of the available QP range to a first QP, a second sub-range extending from a second QP, greater than the first QP, to a third QP, greater than the second QP, and a third sub-range extending from a fourth QP, greater than the third QP, to a greatest available QP of the available QP range, wherein a first quantization matrix for the first sub-range is a flat matrix, a second quantization matrix for the second sub-range comprises a first difference between a first entry corresponding to a DC coefficient and a second entry corresponding to an AC coefficient, and a third quantization matrix for the third sub-range comprises a second difference between a third entry corresponding to the DC coefficient and a fourth entry corresponding to the AC coefficient, the second difference being greater than the first difference.
20. The at least one non-transitory machine readable medium of claim 18 , further comprising instructions that, in response to being executed on the computing device, cause the computing device to perform video coding by:
determining a frame QP for the current frame was generated based on rate control applied using a further quantization matrix; and
adjusting the frame QP to a second frame QP based on a QP adjustment corresponding to the selected quantization matrix and the further quantization matrix, wherein the current frame is encoded using the adjusted frame QP.
21. The at least one non-transitory machine readable medium of claim 18 , further comprising instructions that, in response to being executed on the computing device, cause the computing device to perform video coding by:
determining a second estimated QP, greater than or less than the estimated QP, for a second frame of the video sequence temporally prior to the current frame, the second estimated QP corresponding to a second sub-range of the plurality of sub-ranges and a second quantization matrix; and
encoding the second frame using the selected quantization matrix in response to the estimated QP for the current frame being a quantization matrix switching QP.
22. The at least one non-transitory machine readable medium of claim 18 , wherein determining the estimated quantization parameter for the current frame comprises:
generating a set of features for each of the current frame and the one or more frames subsequent to the current fram;
applying a machine learning model to each of the sets of features to generate a modeled QP for each of the current frame and the one or more frames subsequent to the current frame; and
combining the modeled QPs to determine the estimated QP for the current frame.