Bit-width control for bi-directional optical flow
View Patent ↗A method for controlling bit-width for bi-directional optical flow (BDOF) for video coding includes decoding a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture is displayed before a current picture and the second reference picture is displayed after the current picture. A motion refinement of a coding unit (CU) is calculated by minimizing a difference between a first prediction L 0 and a second prediction L 1 . First gradient values for the first prediction Lo and second gradient values for the second prediction L 1 are calculated. A final bi-prediction of the CU is calculated. Also disclosed are an associated apparatus and an associated non-transitory computer readable storage medium.
1. A method for video decoding, comprising:
decoding a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture is displayed before a current picture and the second reference picture is displayed after the current picture;
determining, for a current block of the current picture, a first prediction L 0 corresponding to the first reference picture;
determining, for the current block, a second prediction L 1 corresponding to the second reference picture;
calculating a respective motion refinement (v x , v y ) of each subblock of the current block by minimizing a difference between the first prediction L 0 and the second prediction L 1 based on a motion refinement threshold and a plurality of internal parameters representing different summation characters of the first prediction L 0 and the second prediction L 1 ;
calculating first gradient values
∂
I
(
0
)
∂
x
(
i
,
j
)
and
∂
I
(
0
)
∂
y
(
i
,
j
)
for the first prediction L 0 and second gradient values
∂
I
(
1
)
∂
x
(
i
,
j
)
and
∂
I
(
1
)
∂
y
(
i
,
j
)
for the second prediction L 1 , wherein I (0) (i, j) is a prediction sample at sample location (i, j) of the first prediction L 0 , and I (1) (i, j) is a prediction sample at the sample location (i, j) of the second prediction L 1 ; and
calculating a final bi-prediction of the current block based on the first prediction L 0 , the second prediction L 1 , respective motion refinements (v x , v y ), the first gradient values
∂
I
(
0
)
∂
x
(
i
,
j
)
and
∂
I
(
0
)
∂
y
(
i
,
j
)
,
and the second gradient values
∂
I
(
1
)
∂
x
(
i
,
j
)
and
∂
I
(
1
)
∂
y
(
i
,
j
)
.
2. The method of claim 1 , wherein a value of (v x , v y ) is calculated based on following equations:
v x =S 1 >0?clip3(− th BDOF ,th BDOF ,−(( S 3 ·2 n c )>>└log 2 S 1 ┘)):0
v y =S 5 >0?clip3(− th BDOF ,th BDOF ,−(( S 6 ·2 n c −(( v x S 2,m )<< n S 2 +v x S 2,s )/2)>>└log 2 S 5 ┘)):0,
wherein th BDOF is the motion refinement threshold,
S
2
,
m
=
S
2
⪢
n
S
2
,
S
2
,
s
=
S
2
&
(
2
n
s
2
-
1
)
,
S 1 , S 2 , S 3 , S 5 and S 6 are the plurality of internal parameters representing different summation characters of the first prediction L 0 and the second prediction L 1 , and n S 2 and n c are integers.
3. The method of claim 2 , wherein a value of S 1 is calculated based on ψ x (i, j) for a set of prediction samples around a respective subblock of each of the first prediction L 0 and the second prediction L 1 , wherein a value of S 2 is calculated based on ψ x (i, j) and ψ y (i, j) for the set of prediction samples, wherein a value of S 3 is calculated based on θ(i, j) and ψ x (i, j) for the set of prediction samples, wherein a value of S 5 is calculated based on ψ y (i, j) for the set of prediction samples, and wherein a value of S 6 is calculated based on θ(i, j) and ψ y (i, j) for the set of prediction samples,
wherein ψ x (i, j), ψ y (i, j) and θ(i, j) are correlation parameters, wherein ψ x (i, j) is calculated based on a first horizontal gradient value of the first prediction L 0 and a second horizontal gradient value of the second prediction L 1 , ψ y (i, j) is calculated based on a first vertical gradient value of the first prediction L 0 and a second vertical gradient value of the second prediction L 1 and θ(i, j) is calculated based on the first prediction L 0 and the second prediction L 1 .
4. The method of claim 2 , wherein a value of n c is set equal to 2.
5. The method of claim 3 , further comprising introducing an additional bit-shift n adj to a calculation of ψ x (i, j), ψ y (i, j), and θ(i, j), wherein:
ψ
x
(
i
,
j
)
=
(
∂
I
(
1
)
∂
x
(
i
,
j
)
+
∂
I
(
0
)
∂
x
(
i
,
j
)
)
⪢
(
3
-
n
adj
)
ψ
y
(
i
,
j
)
=
(
∂
I
(
1
)
∂
y
(
i
,
j
)
+
∂
I
(
0
)
∂
y
(
i
,
j
)
)
⪢
(
3
-
n
adj
)
;
and
θ
(
i
,
j
)
=
(
I
(
1
)
(
i
,
j
)
⪢
(
6
-
n
adj
)
)
-
(
I
(
0
)
(
i
,
j
)
⪢
(
6
-
n
adj
)
)
.
6. The method of claim 5 , wherein a value of n adj is set equal to 2.
7. The method of claim 5 , wherein the value of S 2 is calculated based on ψ x (i, j)·ψ y (i, j) for the set of prediction samples, and wherein the value of S 6 is calculated based on θ(i, j)·ψ y (i, j) for the set of prediction samples.
8. The method of claim 1 , wherein calculating gradient value
∂
I
(
k
)
∂
y
(
i
,
j
)
,
k=0, 1 and gradient value
∂
I
(
k
)
∂
x
(
i
,
j
)
,
k=0, 1 further comprises calculating a horizontal gradient and a vertical gradient at each of a first prediction position of the first prediction L 0 and a second prediction position of the second prediction L 1 .
9. The method of claim 8 , wherein the horizontal gradient is calculated at each of the first prediction position and the second prediction position as:
∂
I
(
k
)
∂
x
(
i
,
j
)
=
(
I
(
k
)
(
i
+
1
,
j
)
-
I
(
k
)
(
i
-
1
,
j
)
)
⪢
(
4
+
n
grad
)
,
k
=
0
,
1
;
and the vertical gradient is calculated at each of the first prediction position and the second prediction position as:
∂
I
(
k
)
∂
y
(
i
,
j
)
=
(
I
(
k
)
(
i
,
j
+
1
)
-
I
(
k
)
(
i
,
j
-
1
)
)
⪢
(
4
+
n
grad
)
,
k
=
0
,
1
.
wherein n grad is an additional right shift.
10. The method of claim 1 , wherein calculating the final bi-prediction of the current block comprises:
calculating the final bi-prediction of the current block based further on an offset value o offset for combining the first prediction L 0 and the second prediction L 1 .
11. The method of claim 1 , wherein the first prediction L 0 is determined based on a first motion vector (MV 0 ) from the current block to the first reference picture, and wherein the second prediction L 1 is determined based on a second motion vector (MV 1 ) from the current block to the second reference picture.
12. A computing device, comprising:
one or more processors;
a non-transitory storage coupled to the one or more processors; and
a plurality of programs stored in the non-transitory storage that, when executed by the one or more processors, cause the computing device to perform acts comprising:
decoding a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture is displayed before a current picture and the second reference picture is displayed after the current picture;
determining, for a current block of the current picture, a first prediction L 0 corresponding to the first reference picture;
determining, for the current block, a second prediction L 1 corresponding to the second reference picture;
calculating a respective motion refinement (v x , v y ) of each subblock of the current block by minimizing a difference between the first prediction L 0 and the second prediction L 1 based on a motion refinement threshold and a plurality of internal parameters representing different summation characters of the first prediction L 0 and the second prediction L 1 ;
calculating first gradient values
∂
I
(
0
)
∂
x
(
i
,
j
)
and
∂
I
(
0
)
∂
y
(
i
,
j
)
for the first prediction L 0 and second gradient values
∂
I
(
1
)
∂
x
(
i
,
j
)
and
∂
I
(
1
)
∂
y
(
i
,
j
)
for the second prediction L 1 , wherein I (0) (i, j) is a prediction sample at sample location (i, j) of the first prediction L 0 , and I (1) (i, j) is a prediction sample at the sample location (i, j) of the second prediction L 1 ; and
calculating a final bi-prediction of the current block based on the first prediction L 0 , the second prediction L 1 , respective motion refinements (v x , v y ), the first gradient values
∂
I
(
0
)
∂
x
(
i
,
j
)
and
∂
I
(
0
)
∂
y
(
i
,
j
)
,
and the second gradient values
∂
I
(
1
)
∂
x
(
i
,
j
)
and
∂
I
(
1
)
∂
y
(
i
,
j
)
.
13. The computing device of claim 12 , wherein a value of (v x , v y ) is calculated based on following equations:
v x =S 1 >0?clip3(− th BDOF ,th BDOF ,−(( S 3 ·2 n c )>>└log 2 S 1 ┘)):0
v y =S 5 >0?clip3(− th BDOF ,th BDOF ,−(( S 6 ·2 n c −(( v x S 2,m )<< n S 2 +v x S 2,s )/2)>>└log 2 S 5 ┘)):0,
wherein th BDOF is the motion refinement threshold,
S
2
,
m
=
S
2
⪢
n
S
2
,
S
2
,
s
=
S
2
&
(
2
n
s
2
-
1
)
,
S 1 , S 2 , S 3 , S 5 and S 6 are the plurality of internal parameters representing different summation characters of the first prediction L 0 and the second prediction L 1 , and n S 2 and n c are integers.
14. The computing device of claim 13 , wherein a value of S 1 is calculated based on ψ x (i, j) for a set of prediction samples around a respective subblock of each of the first prediction L 0 and the second prediction L 1 , wherein a value of S 2 is calculated based on ψ x (i, j) and ψ y (i, j) for the set of prediction samples, wherein a value of S 3 is calculated based on θ(i, j) and ψ x (i, j) for the set of prediction samples, wherein a value of S 5 is calculated based on ψ y (i, j) for the set of prediction samples, and wherein a value of S 6 is calculated based on θ(i, j) and ψ y (i, j) for the set of prediction samples,
wherein ψ x (i, j), ψ y (i, j) and θ(i, j) are correlation parameters, wherein ψ x (i, j) is calculated based on a first horizontal gradient value of the first prediction L 0 and a second horizontal gradient value of the second prediction L 1 , ψ y (i, j) is calculated based on a first vertical gradient value of the first prediction L 0 and a second vertical gradient value of the second prediction L 1 and θ(i, j) is calculated based on the first prediction L 0 and the second prediction L 1 .
15. The computing device of claim 13 , wherein a value of n c is set equal to 2.
16. The computing device of claim 14 , further comprising introducing an additional bit-shift n adj to a calculation of ψ x (i, j), ψ y (i, j), and θ(i, j), wherein:
ψ
x
(
i
,
j
)
=
(
∂
I
(
1
)
∂
x
(
i
,
j
)
+
∂
I
(
0
)
∂
x
(
i
,
j
)
)
⪢
(
3
-
n
a
dj
)
ψ
y
(
i
,
j
)
=
(
∂
I
(
1
)
∂
y
(
i
,
j
)
+
∂
I
(
0
)
∂
y
(
i
,
j
)
)
⪢
(
3
-
n
adj
)
;
and
θ
(
i
,
j
)
=
(
I
(
1
)
(
i
,
j
)
⪢
(
6
-
n
adj
)
)
-
(
I
(
0
)
(
i
,
j
)
⪢
(
6
-
n
adj
)
)
.
17. The computing device of claim 16 , wherein a value of n adj is set equal to 2.
18. The computing device of claim 16 , wherein the value of S 2 is calculated based on ψ(i, j)·ψ y (i, j) for the set of prediction samples, and wherein the value of S 6 is calculated based on θ(i, j)·ψ y (i, j) for the set of prediction samples.
19. The computing device of claim 12 , wherein calculating gradient value
∂
I
(
k
)
∂
x
(
i
,
j
)
,
k=0, 1 and gradient value
∂
I
(
k
)
∂
y
(
i
,
j
)
,
k=0, 1 further comprises calculating a horizontal gradient and a vertical gradient at each of a first prediction position of the first prediction L 0 and a second prediction position of the second prediction L 1 .
20. The computing device of claim 19 , wherein the horizontal gradient is calculated at each of the first prediction position and the second prediction position as:
∂
I
(
k
)
∂
x
(
i
,
j
)
=
(
I
(
k
)
(
i
+
1
,
j
)
-
I
(
k
)
(
i
-
1
,
j
)
)
⪢
(
4
+
n
grad
)
,
k
=
0
,
1
;
and the vertical gradient is calculated at each of the first prediction position and the second prediction position as:
∂
I
(
k
)
∂
y
(
i
,
j
)
=
(
I
(
k
)
(
i
,
j
+
1
)
-
I
(
k
)
(
i
,
j
-
1
)
)
⪢
(
4
+
n
grad
)
,
k
=
0
,
1
,
wherein n grad is an additional right shift.
21. A non-transitory computer readable storage medium storing a plurality of programs for execution by a computing device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, cause the computing device to perform acts comprising:
decoding a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture is displayed before a current picture and the second reference picture is displayed after the current picture;
determining, for a current block of the current picture, a first prediction L 0 corresponding to the first reference picture;
determining, for the current block, a second prediction L 1 corresponding to the second reference picture;
calculating a respective motion refinement (v x , v y ) of each subblock of the current block by minimizing a difference between the first prediction L 0 and the second prediction L 1 based on a motion refinement threshold and a plurality of internal parameters representing different summation characters of the first prediction L 0 and the second prediction L 1 ;
calculating first gradient values
∂
I
(
0
)
∂
x
(
i
,
j
)
and
∂
I
(
0
)
∂
y
(
i
,
j
)
for the first prediction L 0 and second gradient values
∂
I
(
k
)
∂
x
(
i
,
j
)
and
∂
I
(
k
)
∂
y
(
i
,
j
)
for the second prediction L 1 , wherein I (0) (i, j) is a prediction sample at sample location (i, j) of the first prediction L 0 , and I (1) (i, j) is a prediction sample at the sample location (i, j) of the second prediction L 1 ; and
calculating a final bi-prediction of the current block based on the first prediction L 0 , the second prediction L 1 , respective motion refinements (v x , v y ), the first gradient values
∂
I
(
0
)
∂
x
(
i
,
j
)
and
∂
I
(
0
)
∂
y
(
i
,
j
)
,
and the second gradient values
∂
I
(
1
)
∂
x
(
i
,
j
)
and
∂
I
(
1
)
∂
y
(
i
,
j
)
.