Fine-grained intra prediction fusion
Aspects of the disclosure includes methods and apparatuses for video decoding and encoding and a method of processing visual media data. The method for video decoding includes receiving coded information in a bitstream indicating that a current block is predicted based on a combination of a plurality of intra prediction modes. The method includes determining a plurality of intra predictions of the current block based on the respective intra prediction modes, determining a fused prediction of the current block based on a weighted summation of the plurality of intra predictions where the weighted summation is according to respective weights associated with the plurality of intra predictions, and reconstructing the current block based on the fused prediction. Each weight is based on one of a plurality of weighting functions that depends on a sample location (x, y) and the intra prediction mode of the intra prediction associated with the respective weight.
1 . A method for video decoding, the method comprising:
receiving coded information in a bitstream, the coded information indicating that a current block is predicted based on a combination of a plurality of intra prediction modes;
determining a plurality of intra predictions of the current block based on respective intra prediction modes of the plurality of intra prediction modes;
determining a fused prediction of the current block based on a weighted summation of the plurality of intra predictions of the current block, the weighted summation being according to respective weights associated with the plurality of intra predictions, and
reconstructing the current block based on the fused prediction, wherein
each of the weights is based on a respective one of a plurality of weighting functions that depends on a sample location (x, y) and the intra prediction mode of the intra prediction associated with the respective weight,
the plurality of intra predictions includes a first intra prediction s 1 and a second intra prediction s 2 , and
the plurality of weighting functions includes a first weighting function w 1 (x, y) associated with the first intra prediction and a second weighting function w 2 (x, y) associated with the second intra prediction, w 1 (x, y) being different from w 2 (x, y).
2 . The method of claim 1 , wherein
the current block is partitioned into subblocks;
subblock partition information of the current block indicates subblock locations and subblock shapes of the respective subblocks;
the respective one of the plurality of weighting functions is further based on the subblock partition information of the current block;
a weighting function
w
i
R
j
(
x
,
y
)
is associated with an ith intra prediction indicated by an index i and a jth subblock R j indicated by an index j in the subblocks, the index i being from 1 to k that is a number of the plurality of intra predictions, the index j being from 1 to L that is a number of the subblocks; and
w
1
(
x
,
y
)
=
w
1
R
j
(
x
,
y
)
,
(
x
,
y
)
∈
R
j
and
w
2
R
j
(
x
,
y
)
,
(
x
,
y
)
∈
R
j
.
3 . The method of claim 2 , wherein
the plurality of intra prediction modes includes a horizontal angular prediction mode and a vertical angular prediction mode;
s 1 is associated with the horizontal angular prediction mode and s 2 is associated with the vertical angular prediction mode, and k is 2;
the subblock partition information of the current block indicates that the current block is partitioned into a bottom-left triangular subblock R 1 and a top-right triangular subblock R 2 , and L is 2;
w
1
R
1
(
x
,
y
)
is associated with s 1 in R 1 ,
w
2
R
1
(
x
,
y
)
is associated with s 2 in R 1 ,
w
1
R
2
(
x
,
y
)
is associated with s 1 in R 2 , and
w
2
R
2
(
x
,
y
)
is associated with s 2 in R 2 ;
w
1
R
1
(
x
,
y
)
>
w
2
R
1
(
x
,
y
)
;
and
w
1
R
2
(
x
,
y
)
<
w
2
R
2
(
x
,
y
)
.
4 . The method of claim 2 , wherein
the plurality of intra prediction modes includes an angular prediction mode and a non-angular prediction mode;
s 1 is associated with the angular prediction mode and s 2 is associated with the non-angular prediction mode;
the subblock partition information of the current block indicates that the current block is partitioned into an L-shaped top-left subblock R 1 and a rectangular bottom-right subblock R 2 ;
w
1
R
1
(
x
,
y
)
is associated with s 1 in R 1 ,
w
2
R
1
(
x
,
y
)
is associated with s 2 in R 1 ,
w
1
R
2
(
x
,
y
)
is associated with s 1 in R 2 , and
w
2
R
2
(
x
,
y
)
is associated with s 2 in R 2 ;
w
1
R
1
(
x
,
y
)
>
w
2
R
1
(
x
,
y
)
;
and
w
1
R
2
(
x
,
y
)
<
w
2
R
2
(
x
,
y
)
.
5 . The method of claim 2 , wherein
the plurality of intra prediction modes includes a horizontal angular prediction mode, a vertical angular prediction mode, and a non-angular prediction mode;
s 1 is associated with the horizontal angular prediction mode, s 2 is associated with the vertical angular prediction mode, and the plurality of intra predictions includes a third intra prediction s 3 associated with the non-angular prediction mode;
the subblock partition information of the current block indicates that the current block is partitioned into a left subblock R 1 , a top subblock R 2 , and a bottom-right subblock R 3 , R 1 and R 2 being to the left of R 3 and above R 3 , respectively;
the plurality of weighting functions includes a third weighting function w 3 (x, y) associated with s 3 ;
w
3
(
x
,
y
)
=
w
3
R
j
(
x
,
y
)
,
(
x
,
y
)
∈
R
j
;
w
1
R
1
(
x
,
y
)
is associated with s 1 in R 1 ,
w
2
R
1
(
x
,
y
)
is associated with s 2 in R 1 ,
w
3
R
1
(
x
,
y
)
is associated with s 3 in R 1 ,
w
1
R
2
(
x
,
y
)
is associated with s 1 in R 2 ,
w
2
R
2
(
x
,
y
)
is associated with s 2 in R 2 ,
w
3
R
2
(
x
,
y
)
is associated with s 3 in R 2 ,
w
1
R
3
(
x
,
y
)
is associated with s 1 in R 3 ,
w
2
R
3
(
x
,
y
)
is associated with s 2 in R 3 , and
w
3
R
3
(
x
,
y
)
is associated with s 3 in R 3 ;
w
1
R
1
(
x
,
y
)
is
larger
than
w
2
R
1
(
x
,
y
)
and
w
3
R
1
(
x
,
y
)
;
w
2
R
2
(
x
,
y
)
is
larger
than
w
1
R
2
(
x
,
y
)
and
w
3
R
2
(
x
,
y
)
;
and
w
3
R
3
(
x
,
y
)
is
larger
than
w
1
R
3
(
x
,
y
)
and
w
2
R
3
(
x
,
y
)
.
6 . The method of claim 2 , wherein
the plurality of intra prediction modes includes a horizontal angular prediction mode, a vertical angular prediction mode, and a non-angular prediction mode;
s 1 is associated with the horizontal angular prediction mode, s 2 is associated with the vertical angular prediction mode, and the plurality of intra predictions includes a third intra prediction s 3 associated with the non-angular prediction mode;
the subblock partition information of the current block indicates that the current block is partitioned into a top-left subblock R 1 , a bottom-left subblock R 2 , a top-right subblock R 3 , and a bottom-right subblock R 4 ;
the plurality of weighting functions includes a third weighting function w 3 (x, y) associated with s 3 ;
w
3
(
x
,
y
)
=
w
3
R
j
(
x
,
y
)
,
(
x
,
y
)
∈
R
j
;
w
1
R
1
(
x
,
y
)
is associated with s 1 in R 1 ,
w
2
R
1
(
x
,
y
)
is associated with s 2 in R 1 ,
w
3
R
1
(
x
,
y
)
is associated with s 3 in R 1 ,
w
1
R
2
(
x
,
y
)
is associated with s 1 in R 2 ,
w
2
R
2
(
x
,
y
)
is associated with s 2 in R 2 ,
w
3
R
2
(
x
,
y
)
is associated with s 3 in R 2 ,
w
1
R
3
(
x
,
y
)
is associated with s 1 in R 3 ,
w
2
R
3
(
x
,
y
)
is associated with s 2 in R 3 ,
w
3
R
3
(
x
,
y
)
is associated with s 3 in R 3 ,
w
1
R
4
(
x
,
y
)
is associated with s 1 in R 4 ,
w
2
R
4
(
x
,
y
)
is associated with s 2 in R 4 , and
w
3
R
4
(
x
,
y
)
is associated with s 3 in R 4 ;
w
1
R
1
(
x
,
y
)
=
w
2
R
1
(
x
,
y
)
;
w
1
R
2
(
x
,
y
)
is
larger
than
w
2
R
2
(
x
,
y
)
and
w
3
R
2
(
x
,
y
)
;
w
2
R
3
(
x
,
y
)
is
larger
than
w
1
R
3
(
x
,
y
)
and
w
3
R
3
(
x
,
y
)
;
and
w
3
R
4
(
x
,
y
)
is
larger
than
w
1
R
4
(
x
,
y
)
and
w
2
R
4
(
x
,
y
)
.
7 . The method of claim 2 , wherein
the plurality of intra prediction modes includes an angular prediction mode and a non-angular prediction mode;
s 1 is associated with the angular prediction mode and s 2 is associated with the non-angular prediction mode;
the subblock partition information of the current block indicates that the current block is partitioned into an L-shaped top-left subblock R 1 and a rectangular bottom-right subblock R 2 ;
w
1
R
1
(
x
,
y
)
is associated with s 1 in R 1 ,
w
2
R
1
(
x
,
y
)
is associated with s 2 in R 1 ,
w
1
R
2
(
x
,
y
)
is associated with s 1 in R 2 , and
w
1
R
2
(
x
,
y
)
is associated with s 2 in R 2 ;
w
1
R
1
(
x
,
y
)
=
C
1
;
w
2
R
1
(
x
,
y
)
=
C
2
;
W
1
R
2
(
x
,
y
)
is
proportional
to
C
3
(
1
-
x
w
)
or
C
3
(
1
-
y
H
)
;
w
2
R
2
(
x
,
y
)
is proportional to
C
3
(
x
w
)
;
when
w
1
R
2
(
x
,
y
)
is proportional to
C
3
(
1
-
x
w
)
and
w
2
R
2
(
x
,
y
)
is proportional to
C
3
(
y
H
)
when
w
1
R
2
(
x
,
y
)
is proportional to
C
3
(
1
-
y
H
)
;
and C 1 , C 2 are C 3 are constants.
8 . The method of claim 2 , wherein
the plurality of intra prediction modes includes a first angular prediction mode, a second angular prediction mode, and a non-angular prediction mode;
s 1 is associated with the first angular prediction mode, s 2 is associated with the second angular prediction mode, and the plurality of intra predictions includes a third intra prediction s 3 associated with the non-angular prediction mode;
the subblock partition information of the current block indicates that the current block is partitioned into an L-shaped top-left subblock R 1 and a rectangular bottom-right subblock R 2 ;
the plurality of weighting functions includes a third weighting function w 3 (x, y) associated with s 3 ;
w
3
(
x
,
y
)
=
w
3
R
j
(
x
,
y
)
,
(
x
,
y
)
∈
R
j
;
w
1
R
1
(
x
,
y
)
is associated with s 1 in R 1 ,
w
2
R
1
(
x
,
y
)
is associated with s 2 in R 1 ,
w
3
R
1
(
x
,
y
)
is associated with s 3 in R 1 ,
w
1
R
2
(
x
,
y
)
is associated with s 1 in R 2 ,
w
2
R
2
(
x
,
y
)
is associated with s 2 in R 2 ,
w
3
R
2
(
x
,
y
)
is associated with s 3 in R 2 ;
w
1
R
1
(
x
,
y
)
,
w
2
R
1
(
x
,
y
)
,
and
w
3
R
1
(
x
,
y
)
are constants; and
w
1
R
2
(
x
,
y
)
and
w
2
R
2
(
x
,
y
)
decrease with one of x and y, and
w
3
R
2
(
x
,
y
)
increases with the one of x and y.
9 . The method of claim 1 , wherein
the plurality of intra prediction modes includes at least one angular prediction mode and at least one non-angular prediction mode;
w 1 (x, y) that is associated with the at least one angular prediction mode decreases towards one of a right direction and a bottom direction; and
at least one weighting function of the plurality of weighting functions that is associated with the at least one non-angular prediction mode increases towards the one of the right direction and the bottom direction.
10 . The method of claim 9 , wherein
the at least one angular prediction mode is an angular prediction mode, and the at least one non-angular prediction mode is a non-angular prediction mode;
w 1 (x, y) is proportional to
C
1
(
1
-
x
W
)
or
C
1
(
1
-
y
H
)
,
W and H being a width and a height of the current block, respectively;
when w 1 (x, y) is proportional to
C
1
(
1
-
x
W
)
,
the at least one weighting function associated with the non-angular prediction mode is w 2 (x, y) proportional to
C
1
(
1
-
x
W
)
;
and
when w 1 (x, y) is proportional to
C
1
(
1
-
y
H
)
,
w 2 (x, y) is proportional to
C
1
(
y
H
)
.
11 . The method of claim 9 , wherein
the at least one angular prediction mode includes a first angular prediction mode and a second angular prediction mode, and the at least one non-angular prediction mode is a non-angular prediction mode;
w 1 (x, y) of the at least one weighting function associated with the first angular prediction mode is proportional to
C
3
(
1
-
x
W
)
or
C
3
(
1
-
y
H
)
,
W and H being a width and a height of the current block, respectively;
when w 1 (x, y) is proportional to
C
3
(
1
-
x
W
)
,
w 2 (x, y) of the at least one weighting function associated with the second angular prediction mode is proportional to
C
4
(
1
-
x
W
)
,
and a third weighting function w 3 (x, y) of the at least one weighting function associated with the non-angular prediction mode is proportional to
C
5
(
x
W
)
;
and
when w 1 (x, y) is proportional to
C
3
(
1
-
y
H
)
,
w 2 (x, y) is proportional to
C
4
(
1
-
y
H
)
,
and w 3 (x, y) is proportional to
C
5
(
y
H
)
.
12 . A method for video encoding, the method comprising:
determining a plurality of intra predictions of a current block based on a plurality of respective intra prediction modes;
determining a fused prediction of the current block based on a weighted summation of the plurality of intra predictions of the current block, the weighted summation being according to respective weights associated with the plurality of intra predictions, and
encoding the current block based on the fused prediction, wherein
each of the weights is based on a respective one of a plurality of weighting functions that depends on a sample location (x, y) and the intra prediction mode of the intra prediction associated with the respective weight,
the plurality of intra predictions includes a first intra prediction s 1 and a second intra prediction s 2 , and
the plurality of weighting functions includes a first weighting function w 1 (x, y) associated with the first intra prediction and a second weighting function w 2 (x, y) associated with the second intra prediction, w 1 (x, y) being different from w 2 (x, y).
13 . The method of claim 12 , wherein
the current block is partitioned into subblocks;
subblock partition information of the current block indicates subblock locations and subblock shapes of the respective subblocks;
the respective one of the plurality of weighting functions is further based on the subblock partition information of the current block;
a weighting function
w
i
R
j
(
x
,
y
)
is associated with an ith intra prediction indicated by an index i and a jth subblock R j indicated by an index j in the subblocks, the index i being from 1 to k that is a number of the plurality of intra predictions, the index j being from 1 to L that is a number of the subblocks; and
w
1
(
x
,
y
)
=
w
1
R
j
(
x
,
y
)
,
(
x
,
y
)
∈
R
j
and
w
2
(
x
,
y
)
=
w
2
R
j
(
x
,
y
)
,
(
x
,
y
)
∈
R
j
.
14 . The method of claim 13 , wherein
the plurality of intra prediction modes includes a horizontal angular prediction mode and a vertical angular prediction mode;
s 1 is associated with the horizontal angular prediction mode and s 2 is associated with the vertical angular prediction mode, and k is 2;
the subblock partition information of the current block indicates that the current block is partitioned into a bottom-left triangular subblock R 1 and a top-right triangular subblock R 2 , and L is 2;
w
1
R
1
(
x
,
y
)
is associated with s 1 in R 1 ,
w
2
R
1
(
x
,
y
)
is associated with s 2 in R 1 ,
w
1
R
2
(
x
,
y
)
is associated with s 1 in R 2 , and
w
2
R
2
(
x
,
y
)
is associated with s 2 in R 2 ;
w
1
R
1
(
x
,
y
)
>
w
2
R
1
(
x
,
y
)
;
and
w
1
R
2
(
x
,
y
)
<
w
2
R
2
(
x
,
y
)
.
15 . The method of claim 13 , wherein
the plurality of intra prediction modes includes an angular prediction mode and a non-angular prediction mode;
s 1 is associated with the angular prediction mode and s 2 is associated with the non-angular prediction mode;
the subblock partition information of the current block indicates that the current block is partitioned into an L-shaped top-left subblock R 1 and a rectangular bottom-right subblock R 2 ;
w
1
R
1
(
x
,
y
)
is associated with s 1 in R 1 ,
w
2
R
1
(
x
,
y
)
is associated with s 2 in R 1 ,
w
1
R
2
(
x
,
y
)
is associated with s 1 in R 2 , and
w
2
R
2
(
x
,
y
)
is associated with s 2 in R 2 ;
w
1
R
1
(
x
,
y
)
>
w
2
R
1
(
x
,
y
)
;
and
w
1
R
2
(
x
,
y
)
<
w
2
R
1
(
x
,
y
)
.
16 . The method of claim 13 , wherein
the plurality of intra prediction modes includes a horizontal angular prediction mode, a vertical angular prediction mode, and a non-angular prediction mode;
s 1 is associated with the horizontal angular prediction mode, s 2 is associated with the vertical angular prediction mode, and the plurality of intra predictions includes a third intra prediction s 3 associated with the non-angular prediction mode;
the subblock partition information of the current block indicates that the current block is partitioned into a left subblock R 1 , a top subblock R 2 , and a bottom-right subblock R 3 , R 1 and R 2 being to the left of R 3 and above R 3 , respectively;
the plurality of weighting functions includes a third weighting function w 3 (x, y) associated with s 3 ;
w
3
(
x
,
y
)
=
w
3
R
j
(
x
,
y
)
,
(
x
,
y
)
∈
R
j
;
w
1
R
1
(
x
,
y
)
is associated with s 1 in R 1 ,
w
2
R
1
(
x
,
y
)
is associated with s 2 in R 1 ,
w
3
R
1
(
x
,
y
)
is associated with s 3 in R 1 ,
w
1
R
2
(
x
,
y
)
is associated with S 1 in R 2 ;
w
2
R
2
(
x
,
y
)
is associated with s 2 in R 2 ,
w
3
R
2
(
x
,
y
)
is associated with s 3 in R 2 ,
w
1
R
3
(
x
,
y
)
is associated with s 1 in R 3 ,
w
2
R
3
(
x
,
y
)
is associated with s 2 in R 3 , and
w
3
R
3
(
x
,
y
)
is associated with s 3 in R 3 ;
w
1
R
1
(
x
,
y
)
is
larger
than
w
2
R
1
(
x
,
y
)
and
w
3
R
1
(
x
,
y
)
;
w
2
R
2
(
x
,
y
)
is
larger
than
w
1
R
2
(
x
,
y
)
and
w
3
R
2
(
x
,
y
)
;
and
w
3
R
3
(
x
,
y
)
is
larger
than
w
1
R
3
(
x
,
y
)
and
w
2
R
3
(
x
,
y
)
.
17 . The method of claim 13 , wherein
the plurality of intra prediction modes includes a horizontal angular prediction mode, a vertical angular prediction mode, and a non-angular prediction mode;
s 1 is associated with the horizontal angular prediction mode, s 2 is associated with the vertical angular prediction mode, and the plurality of intra predictions includes a third intra prediction s 3 associated with the non-angular prediction mode;
the subblock partition information of the current block indicates that the current block is partitioned into a top-left subblock R 1 , a bottom-left subblock R 2 , a top-right subblock R 3 , and a bottom-right subblock R 4 ;
the plurality of weighting functions includes a third weighting function w 3 (x, y) associated with s 3 ;
w
3
(
x
,
y
)
=
w
3
R
j
(
x
,
y
)
,
(
x
,
y
)
∈
R
j
;
w
1
R
1
(
x
,
y
)
is associated with s 1 in R 1 ,
w
2
R
1
(
x
,
y
)
is associated with s 2 in R 1 ,
w
3
R
1
(
x
,
y
)
is associated with s 3 in R 1 ,
w
1
R
2
(
x
,
y
)
is associated with s 1 in R 2 ,
w
2
R
2
(
x
,
y
)
is associated with s 2 in R 2 ,
w
3
R
2
(
x
,
y
)
is associated with s 3 in R 2 ,
w
1
R
3
(
x
,
y
)
is associated with s 1 in R 3 ;
w
2
R
3
(
x
,
y
)
is associated with s 2 in R 3 ,
w
3
R
3
(
x
,
y
)
is associated with s 3 in R 3 ,
w
1
R
4
(
x
,
y
)
is associated with s 1 in R 4 ,
w
2
R
4
(
x
,
y
)
is associated with s 2 in R 4 , and
w
3
R
4
(
x
,
y
)
is associated with s 3 in R 4 ;
w
1
R
1
(
x
,
y
)
=
w
2
R
1
(
x
,
y
)
;
w
1
R
2
(
x
,
y
)
is
larger
than
w
2
R
2
(
x
,
y
)
and
w
3
R
2
(
x
,
y
)
;
w
2
R
3
(
x
,
y
)
is
larger
than
w
1
R
3
(
x
,
y
)
and
w
3
R
3
(
x
,
y
)
;
and
w
3
R
4
(
x
,
y
)
is
larger
than
w
1
R
4
(
x
,
y
)
and
w
2
R
4
(
x
,
y
)
.
18 . The method of claim 13 , wherein
the plurality of intra prediction modes includes an angular prediction mode and a non-angular prediction mode;
s 1 is associated with the angular prediction mode and s 2 is associated with the non-angular prediction mode;
the subblock partition information of the current block indicates that the current block is partitioned into an L-shaped top-left subblock R 1 and a rectangular bottom-right subblock R 2 ;
w
1
R
1
(
x
,
y
)
is associated with s 1 in R 1 ,
w
2
R
1
(
x
,
y
)
is associated with s 2 in R 1 ,
w
1
R
2
(
x
,
y
)
is associated with s 1 in R 2 , and
w
2
R
2
(
x
,
y
)
is associated with s 2 in R 2 ;
w
1
R
1
(
x
,
y
)
=
C
1
;
w
2
R
1
(
x
,
y
)
=
C
2
;
W
1
R
2
(
x
,
y
)
is
proportional
to
C
3
(
1
-
x
w
)
or
C
3
(
1
-
y
H
)
;
w
1
R
2
(
x
,
y
)
is proportional to
C
3
(
x
w
)
when
w
1
R
2
(
x
,
y
)
is proportional to
C
3
(
1
-
x
w
)
and
w
2
R
2
(
x
,
y
)
is proportional to
C
3
(
y
H
)
when
w
1
R
2
(
x
,
y
)
is proportional to
C
3
(
1
-
y
H
)
;
and
C 1 , C 2 are C 3 are constants.
19 . A non-transitory computer-readable storage medium storing instructions which when executed by a processor cause the processor to perform a method of encoding a bitstream comprising:
determining a plurality of intra predictions of a current block based on a plurality of respective intra prediction modes;
determining a fused prediction of the current block based on a weighted summation of the plurality of intra predictions of the current block, the weighted summation being according to respective weights associated with the plurality of intra predictions;
encoding, in the bitstream, the current block based on the fused prediction; and
transmitting the bitstream, wherein
each of the weights is based on a respective one of a plurality of weighting functions that depends on a sample location (x, y) and the intra prediction mode of the intra prediction associated with the respective weight,
the plurality of intra predictions includes a first intra prediction s 1 and a second intra prediction s 2 , and
the plurality of weighting functions includes a first weighting function w 1 (x, y) associated with the first intra prediction and a second weighting function w 2 (x, y) associated with the second intra prediction, w 1 (x, y) being different from w 2 (x, y).
20 . The method of claim 2 , further comprising:
determining, for the ith intra prediction, the weighting function
w
i
R
j
(
x
,
y
)
for the jth subblock R j and a weighting function
w
i
R
j
+
1
(
x
,
y
)
for the (j+1)th subblock R j+1 separately.