Adapting the transform process to neural network-based intra prediction mode
At least a method and an apparatus are presented for efficiently encoding or decoding video. For example, an intra prediction of an image block using at least one neural network from a context comprising pixels surrounding the image block is determined and an information relative to a transform method to apply for decoding the image block is also determined. The transform method is adapted to the neural network intra prediction mode of the block to encode or decode. The information relative to the transform method is inferred from the at least one neural network used in intra prediction of the image block at the encoding and either signaled or also inferred at the decoding.
1. A method comprising:
determining an intra prediction of an image block and a vector by applying at least one neural network to a context comprising pixels adjacent to the image block, wherein an ith element of the vector indicates a relative relevance that a respective group of transforms corresponding to an index i includes a transform that is applied to the image block and wherein the index i identifies the respective group of transforms from among a plurality of groups of transforms;
inferring an index corresponding to a higher relative relevance of the vector, wherein the index identifies a particular group of transforms from among a plurality of groups of transforms;
obtaining a block of residue for the image block by at least applying an inverse transform to transform coefficients of the image block, the inverse transform belonging to the particular group of transforms identified by the index; and
decoding the image block based on the intra prediction and the block of residue.
2. The method of claim 1 , further comprising:
decoding an explicit transform index, wherein the explicit transform index indicates a particular transform applied to the image block among the particular group of transforms.
3. The method of claim 1 , wherein the at least one neural network comprises one or more input data among:
a Quantization Parameter used to decode some of the pixels that belongs to the context;
an index of an intra prediction mode used to predict the image block that is located at a top-right of the context if the image block at the top-right of the context is predicted in intra; and
an index of an intra prediction mode used to predict an image block that is located at a bottom-left of the context if the image block at the bottom-left of the context is predicted in intra.
4. The method of claim 1 , wherein obtaining a block of residue for the image block further comprises:
applying an inverse secondary transform to the transform coefficients; and
applying an inverse primary transform after the inverse secondary transform to obtain the block of residue, wherein the inverse secondary transform belongs to the particular group of transforms identified by the index.
5. The method of claim 1 , wherein obtaining a block of residue for the image block further comprises:
applying an inverse secondary Low Frequency Non-Separable Transform LFNST to the transform coefficients; and
applying an inverse primary transform to the inverse secondary transform coefficients, wherein a group of transforms includes 2 LFNST matrices with a decision of transposing primary transform coefficients.
6. The method of claim 5 , further comprising:
decoding an index of a Low Frequency Non-Separable Transform in the group of 2 LFNST matrices.
7. An apparatus comprising at least one processor configured to:
determine an intra prediction of an image block and a vector by applying at least one neural network to a context comprising pixels adjacent to the image block, wherein an ith element of the vector indicates a relative relevance that a respective group of transforms corresponding to an index i includes a transform that is applied to the image block and wherein the index i identifies the respective group of transforms from among a plurality of groups of transforms;
infer an index corresponding to a higher relative relevance of the vector, wherein the index identifies a particular group of transforms from among a plurality of groups of transforms;
obtain a block of residue for the image block by at least applying a particular inverse transform to transform coefficients of the image block, the particular inverse transform belonging to the particular group of transforms identified by the index; and
decode the image block based on the intra prediction and the block of residue.
8. The apparatus of claim 7 , the at least one processor further configured to:
decode an explicit transform index, wherein the explicit transform index indicates a particular transform applied to the image block among the particular group of transforms.
9. The apparatus of claim 7 , wherein the at least one neural network comprises one or more input data among:
a Quantization Parameter used to decode some of the pixels that belongs to the context;
an index of an intra prediction mode used to predict the image block that is located at a top-right of the context if the image block at the top-right of the context is predicted in intra; and
an index of an intra prediction mode used to predict an image block that is located at a bottom-left of the context if the image block at the bottom-left of the context is predicted in intra.
10. The apparatus of claim 7 , the at least one processor further configured to:
apply an inverse secondary transform to the transform coefficients; and
apply an inverse primary transform after the inverse secondary transform to obtain the block of residue, wherein the inverse secondary transform belongs to the particular group of transforms identified by the index.
11. The apparatus of claim 7 , the at least one processor further configured to:
apply an inverse secondary Low Frequency Non-Separable Transform LFNST to the transform coefficients; and
apply an inverse primary transform to the inverse secondary transform coefficients to obtain the block of residue, wherein the group of transforms includes 2 LFNST matrices with a decision of transposing primary transform coefficients.
12. The apparatus of claim 11 , the at least one processor further configured to:
decode an index of a Low Frequency Non-Separable Transform in the group of 2 LFNST matrices.
13. A method comprising:
determining an intra prediction of an image block and a vector by applying at least one neural network to a context comprising pixels adjacent to the image block, wherein an ith element of the vector indicates a relative relevance that a respective group of transforms corresponding to an index i includes a transform that is applied to the image block and wherein the index i identifies the respective group of transforms from among a plurality of groups of transforms;
inferring an index corresponding to a higher relative relevance of the vector, wherein the index identifies a particular group of transforms from among a plurality of groups of transforms;
at least applying a particular transform to a block of residue for the image block to obtain transform coefficients of the image block, the particular transform belonging to the particular group of transforms identified by the index; and
encoding the block of transform coefficients.
14. The method of claim 13 , further comprising:
encoding an explicit transform index, wherein the explicit transform index indicates the particular transform applied to the image block among the particular group of transforms.
15. The method of claim 13 , wherein the at least one neural network comprises one or more input data among:
a Quantization Parameter used to encode the image block that either partially or fully belongs to the context;
an index of an intra prediction mode used to predict the image block that is located at a top-right of the context if the image block at the top-right of the context is predicted in intra; and
an index of an intra prediction mode used to predict an image block that is located at a bottom-left of the context if the image block at the bottom-left of the context is predicted in intra.
16. The method of claim 13 , wherein obtaining a block of transform coefficient for the image block further comprises:
applying a primary transform to the block of residue; and
applying a secondary transform after the primary transform to obtain to the block of transform coefficients, wherein the secondary transform belongs to the particular group of transforms identified by the index.
17. The method of claim 13 , wherein obtaining a block of transform coefficient for the image block further comprises:
applying a primary transform to the block of residue; and
applying a secondary Low Frequency Non-Separable Transform LFNST to the block of primary transform coefficients, wherein a group of transforms includes 2 LFNST matrices with a decision of transposing primary transform coefficients.
18. The method of claim 17 , further comprising:
encoding an index of a Low Frequency Non-Separable Transform in the group of 2 LFNST matrices.
19. An apparatus comprising at least one processor configured to:
determine an intra prediction of an image block and a vector by applying least one neural network applied to a context comprising pixels adjacent to the image block, wherein an ith element of the vector indicates a relative relevance that a respective group of transforms corresponding to an index i includes a transform that is applied to the image block and wherein the index i identifies the respective group of transforms from among a plurality of groups of transforms;
infer an index corresponding to a higher relative relevance of the vector, wherein the index identifies a particular group of transforms from among a plurality of groups of transforms;
at least apply a particular transform to a block of residue to obtain transform coefficients of the image block, the particular transform belonging to the particular group of transforms identified by the index; and
encode the block of transform coefficients.
20. The apparatus of claim 19 , the at least one processor further configured to:
encode an explicit transform index, wherein the explicit transform index indicates the particular transform applied to the image block among the particular group of transforms.
21. The apparatus of claim 19 , wherein the at least one neural network comprises one or more input data among:
a Quantization Parameter used to encode the image block that either partially or fully belongs to the context;
an index of an intra prediction mode used to predict the image block that is located at a top-right of the context if the image block at the top-right of the context is predicted in intra; and
an index of an intra prediction mode used to predict an image block that is located at a bottom-left of the context if the image block at the bottom-left of the context is predicted in intra.
22. The apparatus of claim 19 , the at least one processor further configured to:
apply a primary transform to the block of residue; and
apply a secondary transform after the primary transform to obtain to the block of transform coefficients, wherein the secondary transform belongs to the particular group of transforms identified by the index.
23. The apparatus of claim 19 , the at least one processor further configured to:
apply a primary transform to the block of residue; and
apply a secondary Low Frequency Non-Separable Transform LFNST to the block of primary transform coefficients, wherein the group of transforms includes 2 LFNST matrices with a decision of transposing primary transform coefficients.
24. The apparatus of claim 23 , the at least one processor further configured to encode an index of a Low Frequency Non-Separable Transform in the group of 2 LFNST matrices.