Ultra light models and decision fusion for fast video coding
Ultra light models and decision fusion for increasing the speed of intra-prediction are described. Using a machine-learning (ML) model, an ML intra-prediction mode is obtained. A most-probable intra-prediction mode is obtained from amongst available intra-prediction modes for encoding the current block. As an encoding intra-prediction mode, one of the ML intra-prediction mode or the most-probable intra-prediction mode is selected, and the encoding intra-prediction mode is encoded in a compressed bitstream. A current block is encoded using the encoding intra-prediction mode. Selection of the encoding intra-prediction mode is based on relative reliabilities of the ML intra-prediction mode and the most-probable intra-prediction mode.
1. A method for encoding a current block of video using intra-prediction, comprising:
obtaining, using a machine-learning (ML) model, an ML intra-prediction mode;
obtaining a most-probable intra-prediction mode from amongst available intra-prediction modes for encoding the current block;
selecting, as an encoding intra-prediction mode, one of the ML intra-prediction mode or the most-probable intra-prediction mode, wherein the selecting is based on relative reliabilities of the ML intra-prediction mode and the most-probable intra-prediction mode, wherein the relative reliabilities are values of a decision matrix generated during training of the ML model;
encoding, in a compressed bitstream, the encoding intra-prediction mode; and
encoding the current block using the encoding intra-prediction mode.
2. The method of claim 1 , wherein the one of the ML intra-prediction mode or the most-probable intra-prediction mode is more reliable than the other of the one of the ML intra-prediction mode or the most-probable intra-prediction mode in case where the one of the ML intra-prediction mode or the most-probable intra-prediction mode provides the best rate-distortion performance.
3. The method of claim 1 , wherein, in a training phase, the ML model is trained using input data to output the ML intra-prediction mode for the current block, and an input datum of the input data comprises an input block of video and a corresponding ground-truth intra-prediction mode for encoding the input block of video.
4. The method of claim 3 , wherein selecting, as the encoding intra-prediction mode, one of the ML intra-prediction mode or the most-probable intra-prediction mode comprises:
using the decision matrix to select the encoding intra-prediction mode, wherein:
the decision matrix is generated during the training phase using statistics that compare respective performances of the ML model and an encoder selecting most-probable intra-prediction modes as compared to ground-truth intra prediction modes, and
the decision matrix indicating whether the ML intra-prediction mode or the most-probable intra-prediction mode is to be selected as the encoding intra-prediction mode.
5. The method of claim 1 , wherein obtaining, using the ML model, the ML intra-prediction mode comprises:
using at least one of a mean of the current block or a variance of the current block as input to the ML model.
6. The method of claim 1 , wherein obtaining, using the ML model, the ML intra-prediction mode comprises:
applying a kernel to the current block to obtain a convolution block;
obtaining a scalar from the convolution block; and
using the scalar as an input to the ML model.
7. The method of claim 6 , wherein the kernel is a Sobel filter corresponding to a direction associated with an available intra-prediction mode of the available intra-prediction modes, and the scalar is a sum of absolute values of the convolution block resulting from applying the Sobel filter.
8. The method of claim 6 , wherein the kernel is applied to the current block using one or more convolution layers that are trained during a training phase.
9. The method of claim 1 , wherein obtaining, using the ML model, the ML intra-prediction mode comprises using a non-linear function of a quantization parameter as an input to the ML model.
10. The method of claim 1 , wherein obtaining, using the ML model, the ML intra-prediction mode comprises:
calculating, for multiple modes of the available intra-prediction modes, respective prediction blocks;
calculating respective errors between the respective prediction blocks and the current block; and
using the respective errors as inputs to the ML model.
11. The method of claim 1 , wherein selecting, as the encoding intra-prediction mode, one of the ML intra-prediction mode or the most-probable intra-prediction mode comprises:
calculating a first cost associated with the ML intra-prediction mode;
selecting the ML intra-prediction mode as the encoding intra-prediction mode; and
reducing the first cost to a second cost.
12. A method for encoding a current block of video using intra-prediction, comprising:
obtaining pre-calculated features, wherein the pre-calculated features comprise at least three of:
a non-linear function of a quantization parameter;
respective errors between the current block and respective prediction blocks, wherein each prediction block corresponds to an available intra-prediction mode;
a mean of the current block;
a variance of the current block; or
a sum-of-absolute values of a convolution block, the convolution block obtained from the current block;
obtaining, using a machine-learning (ML) model, an ML intra-prediction mode, wherein the ML model receives the pre-calculated features as inputs;
selecting an encoding intra-prediction mode using at least the ML intra-prediction mode, wherein selecting the encoding intra-prediction mode using at least the ML intra-prediction mode comprises:
obtaining a most-probable intra-prediction mode from amongst available intra-prediction modes for encoding the current block; and
using a decision matrix to select the encoding intra-prediction mode, wherein the encoding intra-prediction mode is one of the ML intra-prediction mode or the most-probable intra-prediction mode;
encoding, in a compressed bitstream, the encoding intra-prediction mode; and
encoding the current block using the encoding intra-prediction mode.
13. The method of claim 12 , wherein the pre-calculated features comprise the non-linear function of the quantization parameter, the respective errors between the current block and respective prediction blocks, the mean of the current block, the variance of the current block, and the sum-of absolute values of the convolution block.
14. The method of claim 12 , wherein:
the decision matrix is generated during a training phase using statistics that compare respective performances of the ML model and an encoder selecting most-probable intra-prediction modes as compared to ground-truth intra prediction modes, and
the decision matrix indicating whether the ML intra-prediction mode or the most-probable intra-prediction mode is to be selected as the encoding intra-prediction mode.
15. The method of claim 12 , wherein, in a training phase, the ML model is trained using input data to output the ML intra-prediction mode for the current block, wherein an input datum of the input data comprises an input block of video and a best intra-prediction mode for encoding the input block of video.
16. The method of claim 12 , wherein calculating the sum-of-absolute values of the convolution block obtained from the current block comprises applying a Sobel filter to the current block to obtain the convolution block.
17. The method of claim 12 , wherein calculating the sum-of-absolute values of the convolution block obtained from the current block comprises obtaining the convolution block using convolutional operations.
18. An apparatus for encoding a current block of video using intra-prediction, wherein the apparatus:
obtains, using a machine-learning (ML) model, ML intra-prediction modes;
obtains most-probable intra-prediction modes from amongst available intra-prediction modes for encoding the current block;
selects, as an encoding intra-prediction mode, one of the ML intra-prediction modes or one of the most-probable intra-prediction modes, wherein the selecting is based on relative reliabilities of the ML intra-prediction mode and the most-probable intra-prediction mode, wherein the relative reliabilities are values of a decision matrix generated during training of the ML model;
encodes, in a compressed bitstream, the encoding intra-prediction mode; and
encodes the current block using the encoding intra-prediction mode.
19. The apparatus of claim 18 , wherein:
the apparatus selects, as the encoding intra-prediction mode, one of the ML intra-prediction modes or one of the most-probable intra-prediction mode using a decision matrix,
the decision matrix is generated during a training phase of the ML model using statistics that compare respective performances of the ML model and an encoder selecting most-probable intra-prediction modes as compared to ground-truth intra prediction modes, and
the decision matrix indicates whether an ML intra-prediction mode or a most-probable intra-prediction mode is to be selected as the encoding intra-prediction mode.