Applications of template matching with fusion techniques in video coding
Methods and systems are described for intra-prediction using template matching (TM) in video coding. The proposed methods include adaptive fusion when using template-based intra mode derivation using the most probable modes (TIMD), and fusion in intra mode prediction with template matching (Intra TMP).
1 . A method for intra prediction using template matching, the method comprising:
accessing a current block and a template region of the current block, the template region comprising a top template and a left template;
determining a best matching block for the current block using intra prediction with template matching, wherein T1 and L1 denote the top and left-template matching costs between the current block and the best matching block, and P1 denotes corresponding intra-prediction data;
determining a second-best matching block for the current block using intra prediction with template matching, wherein T2 and L2 denote the top and left-template matching costs between the current block and the second-best matching block, and P2 denotes corresponding intra-prediction data;
setting a fusion flag to true if (T1<T2 && L1<L2 && (M1!=DC∥M1!=Planar∥M2!=DC∥M2!=Planar)), wherein,
if the fusion flag is true, performing adaptive fusion comprising:
determining fusion weights for one or more pixels in the current block based on a size of the current block and a position associated with each of the one or more pixels; and
applying the fusion weights to the P1 and P2 data to generate fused intra-prediction data P as a weighted sum of the P1 and the P2 data, wherein M1 and M2 denote the best and second best template-based derived intra-modes using a most probable mode (TIMD), and DC and Planar denote intra modes.
2 . A method for intra prediction using template matching, the method comprising:
accessing a current block and a template region of the current block, the template region comprising a top template and a left template;
determining a best matching block for the current block using intra prediction with template matching, wherein T1 and L1 denote the top and left-template matching costs between the current block and the best matching block, and P1 denotes corresponding intra-prediction data;
determining a second-best matching block for the current block using intra prediction with template matching, wherein T2 and L2 denote the top and left-template matching costs between the current block and the second-best matching block, and P2 denotes corresponding intra-prediction data;
setting a fusion flag to true if (T1<T2 and L1<L2), wherein,
if the fusion flag is true, performing adaptive fusion comprising:
dividing the current block into an upper-right region of pixels (RC3), a lower-left region of pixels (RC1), and a mid-region of pixels (RC2) in between the lower-left region and the upper-right region of pixels;
if T1<T2 and L1>L2:
determining weights for the RC1 region to be lower than 0.5;
determining weights for the RC2 region to be close to 0.5; and
determining weights for the RC3 region to be larger than 0.5; and
generating fused intra-prediction data P as P=(Wadp*P1+(1−Wadp)*P2);
else, if T1>T2 and L1<L2:
determining weights for the RC1 region to be larger than 0.5;
determining weights for the RC2 region to be close to 0.5; and
determining weights for the RC3 region to be lower than 0.5, and
generating fused intra-prediction data P as P=(Wadp*P2+(1−Wadp)*P1), wherein Wadp denotes the fusion weights in each of the RC1, RC2, and RC3 regions.
3 . A method for intra prediction using template matching, the method comprising:
accessing a current block and a template region of the current block, the template region comprising a top template and a left template;
determining a best matching block for the current block using intra prediction with template matching, wherein T1 and L1 denote the top and left-template matching costs between the current block and the best matching block, and P1 denotes corresponding intra-prediction data;
determining a second-best matching block for the current block using intra prediction with template matching, wherein T2 and L2 denote the top and left-template matching costs between the current block and the second-best matching block, and P2 denotes corresponding intra-prediction data;
setting a fusion flag to true if (T1<T2 and L1<L2), wherein,
if the fusion flag is true, performing adaptive fusion comprising
determining fusion weights for one or more pixels in the current block,
wherein for a current block of width size w and height size h, determining the fusion weights comprises computing:
wL
(
i
,
j
)
=
0.5
-
cW
*
j
+
cH
*
i
,
wT
(
i
,
j
)
=
0.5
+
cW
*
j
-
cH
*
i
,
wherein cH=0.25/h, cW=0.25/w, and indices j and i vary from 0 to w−1 and 0 to h−1;
if the fusion flag is true:
if (T1<T2 and L1>L2), then
generating fused intra-prediction data P as P(i,j)=(wT(i,j)*P1+wL(i,j)*P2);
else if (T1>T2 and L1<L2), then
generating the fused intra-prediction data P as P(i,j)=(wL(i,j)*P1+wT(i,j)*P2).
4 . The method of claim 3 , wherein determining the fusion weights comprises computing:
wL
(
i
,
j
)
=
1
/
2
+
(
1
/
(
4
*
(
h
-
i
)
)
)
-
(
1
/
(
4
*
(
w
-
j
)
)
)
,
wT
(
i
,
j
)
=
1
/
2
-
(
1
/
(
4
*
(
h
-
i
)
)
)
+
(
1
/
(
4
*
(
w
-
j
)
)
)
.
5 . The method of claim 3 , wherein determining the fusion weights comprises computing:
wL
(
i
,
j
)
=
0.5
*
(
1
/
2
+
(
1
/
(
4
*
(
h
-
i
)
)
)
-
(
1
/
(
4
*
(
w
-
j
)
)
)
)
+
0.25
,
wT
(
i
,
j
)
=
0.5
*
(
1
/
2
-
(
1
/
(
4
*
(
h
-
i
)
)
)
+
(
1
/
(
4
*
(
w
-
j
)
)
)
)
+
0.25
.
6 . A method for intra prediction using template matching, the method comprising:
accessing a current block and a template region of the current block, the template region comprising a top template and a left template;
determining a best matching block for the current block using intra prediction with template matching, wherein T1 and L1 denote the top and left-template matching costs between the current block and the best matching block, and P1 denotes corresponding intra-prediction data;
determining a second-best matching block for the current block using intra prediction with template matching, wherein T2 and L2 denote the top and left-template matching costs between the current block and the second-best matching block, and P2 denotes corresponding intra-prediction data;
setting a fusion flag to true if (T1<T2 and L1<L2), wherein,
if the fusion flag is true, performing adaptive fusion comprising
determining fusion weights for one or more pixels in the current block, wherein
determining the fusion weights comprises:
dividing the current block into M×N subblocks (PB(i,j)), wherein M denotes the number of horizontal subblocks and N denotes the number of vertical subblocks;
dividing the left template region of the current block into N left sub-templates (L(j));
dividing the top template region of the current block into M top sub-templates (T(i));
for each subblock (PB(i,j)), using its corresponding top sub-template (T(i)) and left sub-template (L(j)):
computing corresponding first sub-cost (CostPB1(i,j)) for the best matching block;
computing corresponding second sub-cost (CostPB2(i,j)) for the second-best matching block; and
deriving local adaptive fusion weights for each subblock:
w
1
=
Cost
PB
2
(
i
,
j
)
/
(
Cost
PB
1
(
i
,
j
)
+
Cost
PB
2
(
i
,
j
)
)
,
w
2
=
1
-
w
1.
7 . The method of claim 6 , wherein
Cost
PB
1
(
i
,
j
)
=
Cost
T
1
(
i
)
+
Cost
L
1
(
j
)
,
and
Cost
PB
2
(
i
,
j
)
=
Cost
T
2
(
i
)
+
Cost
L
2
(
j
)
,
wherein CostT1(i) and CostL1(j) denote the template matching costs with the best matching block when using sub-templates T(j) and L(j), and CostT2(i) and CostL2(j) denote the template matching costs with the second-best matching block when using sub-templates T(j) and L(j).
8 . The method of claim 6 , wherein applying the fusion weights to the PB1 and PB2 data to generate fused intra-prediction data PB as a weighted sum of the PB1 and the PB2 data comprises computing:
PB
(
i
,
j
)
=
w
1
*
PB
1
(
i
,
j
)
+
w
2
*
PB
2
(
i
,
j
)
,
for
i
=
0
,
1
,
…
,
M
-
1
,
and
j
=
0
,
1
,
…
,
N
-
1.
wherein PB1(i, j) and PB2(i,j) denote blocks in P1 and P2 corresponding to PB(i,j).
9 . A method for intra prediction using template matching, the method comprising:
accessing a current block and a template region of the current block, the template region comprising a top template and a left template;
determining a best matching block for the current block using intra prediction with template matching, wherein T1 and L1 denote the top and left-template matching costs between the current block and the best matching block, BMV1 denotes a corresponding best motion vector, and P1 denotes corresponding intra-prediction data using BMVi;
determining a second-best matching block for the current block using intra prediction with template matching, wherein T2 and L2 denote the top and left-template matching costs between the current block and the second-best matching block, BMV2 denotes a corresponding second-best motion vector, and P2 denotes corresponding intra-prediction data using BMV2;
setting a fusion flag to true if (T1<T2 and L1<L2), wherein,
if the fusion flag is true, performing adaptive fusion comprising:
dividing the current block into an upper-right region of pixels (RC3), a lower-left region of pixels (RCi), and a mid-region of pixels (RC2) in between the lower-left region and the upper-right region of pixels;
if T1<T2 and L1>L2:
determining weights for the RCi region to be lower than 0.5;
determining weights for the RC2 region to be close to 0.5; and
determining weights for the RC3 region to be larger than 0.5; and
generating fused intra-prediction data Pas P=(Wadp*Pi+(1−Wadp)*P2);
else, if T1>T2 and L1<L2:
determining weights for the RCi region to be larger than 0.5;
determining weights for the RC2 region to be close to 0.5; and
determining weights for the RC3 region to be lower than 0.5, and
generating fused intra-prediction data P as P=(Wadp*P2+(1−Wadp)*P1), wherein Wadp denotes the weights in each of the RC1, RC2, and RC3 regions.
10 . A method for intra prediction using template matching, the method comprising:
accessing a current block and a template region of the current block, the template region comprising a top template and a left template;
determining a best matching block for the current block using intra prediction with template matching, wherein T1 and L1 denote the top and left-template matching costs between the current block and the best matching block, BMV1 denotes a corresponding best motion vector, and P1 denotes corresponding intra-prediction data using BMV1;
determining a second-best matching block for the current block using intra prediction with template matching, wherein T2 and L2 denote the top and left-template matching costs between the current block and the second-best matching block, BMV2 denotes a corresponding second-best motion vector, and P2 denotes corresponding intra-prediction data using BMV2;
setting a fusion flag to true if (T1<T2 and L1<L2), wherein,
if the fusion flag is true, performing adaptive fusion comprising:
determining weights for one or more pixels in the current block,
wherein for a current block of with size w and height size h, determining weights comprises computing:
wL
(
i
,
j
)
=
0.5
-
cW
*
j
+
cH
*
i
,
wT
(
i
,
j
)
=
0.5
+
cW
*
j
-
cH
*
i
,
wherein cH=0.25/h, cW=0.25/w, and indices j and i vary from 0 to w−1 and 0 to h−1;
if the fusion flag is true:
if (T1<T2 and L1>L2), then
generating fused intra-prediction data P as P(i,j)=(wT(i,j)*P1+wL(i,j)*P2);
else if (T1>T2 and L1<L2), then
generating the fused intra-prediction data P as P(i,j)=(wL(i,j)*P1+wT(i,j)*P2).
11 . The method of claim 10 , wherein determining the weights comprises computing:
wL
(
i
,
j
)
=
1
/
2
+
(
1
/
(
4
*
(
h
-
i
)
)
)
-
(
1
/
(
4
*
(
w
-
j
)
)
)
,
wT
(
i
,
j
)
=
1
/
2
-
(
1
/
(
4
*
(
h
-
i
)
)
)
+
(
1
/
(
4
*
(
w
-
j
)
)
)
.
12 . The method of claim 10 , wherein determining the weights comprises computing:
wL
(
i
,
j
)
=
0.5
*
(
1
/
2
+
(
1
/
(
4
*
(
h
-
i
)
)
)
-
(
1
/
(
4
*
(
w
-
j
)
)
)
)
+
0.25
,
wT
(
i
,
j
)
=
0.5
*
(
1
/
2
-
(
1
/
(
4
*
(
h
-
i
)
)
)
+
(
1
/
(
4
*
(
w
-
j
)
)
)
)
+
0.25
.
13 . A tangible computer-readable storage medium having stored thereon computer-executable instructions for executing with one or more processors a method in accordance with claim 1 .
14 . An apparatus comprising a processor and configured to perform the method recited in claim 1 .