Bit-width control for bi-directional optical flow
View Patent ↗A method for controlling bit-width for bi-directional optical flow (BDOF) for video coding includes acquiring, for a current picture, a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, where the first reference picture is displayed before the current picture and the second reference picture is displayed after the current picture. A motion refinement of a coding unit (CU) is calculated by minimizing a difference between a first prediction L 0 and a second prediction L 1 . First gradient values for the first prediction L 0 and second gradient values for the second prediction L 1 are calculated. A final bi-prediction of the CU is calculated. Also disclosed are an associated apparatus and an associated non-transitory computer readable storage medium.
1. A method for video encoding, comprising:
acquiring, for a current picture, a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture is displayed before the current picture and the second reference picture is displayed after the current picture;
determining, for a current block of the current picture, a first prediction L 0 corresponding to the first reference picture;
determining, for the current block, a second prediction L 1 corresponding to the second reference picture;
calculating a respective motion refinement (v x , v y ) of each subblock of the current block by minimizing a difference between the first prediction L 0 and the second prediction L 1 based on a motion refinement threshold and a plurality of internal parameters representing different summation characters of the first prediction L 0 and the second prediction L 1 ;
calculating first gradient values
∂
I
(
0
)
∂
x
(
i
,
j
)
and
∂
I
(
0
)
∂
y
(
i
,
j
)
for the first prediction L 0 and second gradient values
∂
I
(
1
)
∂
x
(
i
,
j
)
and
∂
I
(
1
)
∂
y
(
i
,
j
)
for the second prediction L 1 , wherein I (0) (i, j) is a prediction sample at sample location (i, j) of the first prediction L 0 , and I (1) (i, j) is a prediction sample at the sample location (i, j) of the second prediction L 1 ; and
calculating a final bi-prediction of the current block based on the first prediction L 0 , the second prediction L 1 , respective motion refinements (v x , v y ), the first gradient values
∂
I
(
0
)
∂
x
(
i
,
j
)
and
∂
I
(
0
)
∂
y
(
i
,
j
)
,
and the second gradient values
∂
I
(
1
)
∂
x
(
i
,
j
)
and
∂
I
(
1
)
∂
y
(
i
,
j
)
.
2. The method of claim 1 , wherein a value of (v x , v y ) is calculated based on following equations:
v x =S 1 >0?clip3(− th BDOF ,th BDOF ,−(( S 3 ·2 n c )>>└log 2 S 1 ┘)):0
v y =S 5 >0?clip3(− th BDOF ,th BDOF ,−(( S 6 ·2 n c −(( v x S 2,m )<< n S 2 +v x S 2,s )/2)>>└log 2 S 5 ┘)):0,
wherein th BDOF is the motion refinement threshold, S 2,m =S 2 >>n S 2 ,
S
2
,
s
=
S
2
&
(
2
n
S
2
-
1
)
,
S 1 , S 2 , S 3 , S 5 and S 6 are the plurality of internal parameters representing different summation characters of the first prediction L 0 and the second prediction L 1 , and n S 2 and n c are integers.
3. The method of claim 2 , wherein a value of S 1 is calculated based on ψ x (i, j) for a set of prediction samples around a respective subblock of each of the first prediction L 0 and the second prediction L 1 , wherein a value of S 2 is calculated based on ψ x (i, j) and ψ y (i, j) for the set of prediction samples, wherein a value of S 3 is calculated based on θ(i, j) and ψ x (i, j) for the set of prediction samples, wherein a value of S 5 is calculated based on ψ y (i, j) for the set of prediction samples, and wherein a value of S 6 is calculated based on θ(i, j) and ψ y (i, j) for the set of prediction samples,
wherein ψ x (i, j), ψ y (i, j) and θ(i, j) are correlation parameters, wherein ψ x (i, j) is calculated based on a first horizontal gradient value of the first prediction L 0 and a second horizontal gradient value of the second prediction L 1 , ψ y (i, j) is calculated based on a first vertical gradient value of the first prediction L 0 and a second vertical gradient value of the second prediction L 1 , and θ(i, j) is calculated based on the first prediction L 0 and the second prediction L 1 .
4. The method of claim 2 , wherein a value of n c is set equal to 2.
5. The method of claim 3 , further comprising introducing an additional bit-shift n adj to calculate ψ x (i, j), ψ y (i, j), and θ(i, j), wherein:
ψ
x
(
i
,
j
)
=
(
∂
I
(
1
)
∂
x
(
i
,
j
)
+
∂
I
(
0
)
∂
x
(
i
,
j
)
)
≫
(
3
-
n
a
d
j
)
;
ψ
y
(
i
,
j
)
=
(
∂
I
(
1
)
∂
y
(
i
,
j
)
+
∂
I
(
0
)
∂
y
(
i
,
j
)
)
≫
(
3
-
n
adj
)
;
and
θ
(
i
,
j
)
=
(
I
(
1
)
(
i
,
j
)
≫
(
6
-
n
adj
)
)
-
(
I
(
0
)
(
i
,
j
)
≫
(
6
-
n
adj
)
)
.
6. The method of claim 5 , wherein a value of n adj is set equal to 2.
7. The method of claim 5 , wherein the value of S 2 is calculated based on ψ x (i, j)·ψ y (i, j) for the set of prediction samples, and wherein the value of S 6 is calculated based on θ(i, j)·ψ y (i, j) for the set of prediction samples.
8. The method of claim 1 , wherein calculating gradient value
∂
I
(
k
)
∂
x
(
i
,
j
)
,
k=0, 1 and gradient value
∂
I
(
k
)
∂
y
(
i
,
j
)
,
k=0, 1 further comprises calculating a horizontal gradient and a vertical gradient at each of a first prediction position of the first prediction L 0 and a second prediction position of the second prediction L 1 .
9. The method of claim 8 , wherein the horizontal gradient is calculated at each of the first prediction position and the second prediction position as:
∂
I
(
k
)
∂
x
(
i
,
j
)
=
(
I
(
k
)
(
i
+
1
,
j
)
-
I
(
k
)
(
i
-
1
,
j
)
)
⪢
(
4
+
n
grad
)
,
k
=
0
,
1
;
and the vertical gradient is calculated at each of the first prediction position and the second prediction position as:
∂
I
(
k
)
∂
y
(
i
,
j
)
=
(
I
(
k
)
(
i
,
j
+
1
)
-
I
(
k
)
(
i
,
j
-
1
)
)
⪢
(
4
+
n
grad
)
,
k
=
0
,
1
,
wherein n grad is an additional right shift.
10. The method of claim 1 , wherein calculating the final bi-prediction of the current block comprises:
calculating the final bi-prediction of the current block based further on an offset value o offset for combining the first prediction L 0 and the second prediction L 1 .
11. The method of claim 1 , wherein the first prediction L 0 is determined based on a first motion vector (MV 0 ) from the current block to the first reference picture, and wherein the second prediction L 1 is determined based on a second motion vector (MV 1 ) from the current block to the second reference picture.
12. A computing device, comprising:
one or more processors;
a non-transitory storage coupled to the one or more processors; and
a plurality of programs stored in the non-transitory storage that, when executed by the one or more processors, cause the computing device to perform acts comprising:
acquiring, for a current picture, a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture is displayed before the current picture and the second reference picture is displayed after the current picture;
determining, for a current block of the current picture, a first prediction L 0 corresponding to the first reference picture;
determining, for the current block, a second prediction L 1 corresponding to the second reference picture;
calculating a respective motion refinement (v x , v y ) of each subblock of the current block by minimizing a difference between the first prediction L 0 and the second prediction L 1 based on a motion refinement threshold and a plurality of internal parameters representing different summation characters of the first prediction L 0 and the second prediction L 1 ;
calculating first gradient values
∂
I
(
0
)
∂
x
(
i
,
j
)
and
∂
I
(
0
)
∂
y
(
i
,
j
)
for the first prediction L 0 and second gradient values
∂
I
(
1
)
∂
x
(
i
,
j
)
and
∂
I
(
1
)
∂
y
(
i
,
j
)
for the second prediction L 1 , wherein I (0) (i, j) is a prediction sample at sample location (i, j) of the first prediction L 0 , and I (1) (i, j) is a prediction sample at the sample location (i, j) of the second prediction L 1 ; and
calculating a final bi-prediction of the current block based on the first prediction L 0 , the second prediction L 1 , respective motion refinements (v x , v y ), the first gradient values
∂
I
(
0
)
∂
x
(
i
,
j
)
and
∂
I
(
0
)
∂
y
(
i
,
j
)
,
and the second gradient values
∂
I
(
1
)
∂
x
(
i
,
j
)
and
∂
I
(
1
)
∂
y
(
i
,
j
)
.
13. The computing device of claim 12 , wherein a value of (v x , v y ) is calculated based on following equations:
v x =S 1 >0?clip3(−th BDOF ,th BDOF ,−(( S 3 ·2 n c )>>└log 2 S 1 ┘)):0
v y =S 5 >0?clip3(−th BDOF ,th BDOF ,−(( S 6 ·2 n c −(( v x S 2,m )<<n S 2 +v x S 2,s )/2)>>└log 2 S 5 ┘)):0,
wherein th BDOF is the motion refinement threshold, S 2,m =S 2 >>n S 2 ,
S
2
,
s
=
S
2
&
(
2
n
S
2
-
1
)
,
S 1 , S 2 , S 3 , S 5 and S 6 are the plurality of internal parameters representing different summation characters of the first prediction L 0 and the second prediction L 1 , and n S 2 , and n c are integers.
14. The computing device of claim 13 , wherein a value of S 1 is calculated based on ψ x (i, j) for a set of prediction samples around a respective subblock of each of the first prediction L 0 and the second prediction L 1 , wherein a value of S 2 is calculated based on ψ x (i, j) and ψ y (i, j) for the set of prediction samples, wherein a value of S 3 is calculated based on θ(i, j) and θ(i, j) for the set of prediction samples, wherein a value of S 5 is calculated based on ψ y (i, j) for the set of prediction samples, and wherein a value of S 6 is calculated based on θ(i, j) and ψ y (i, j) for the set of prediction samples,
wherein ψ(i, j), ψ y (i, j) and θ(i, j) are correlation parameters, wherein ψ x (i, j) is calculated based on a first horizontal gradient value of the first prediction L 0 and a second horizontal gradient value of the second prediction L 1 , ψ y (i, j) is calculated based on a first vertical gradient value of the first prediction L 0 and a second vertical gradient value of the second prediction L 1 , and θ(i, j) is calculated based on the first prediction L 0 and the second prediction L 1 .
15. The computing device of claim 13 , wherein a value of n c is set equal to 2.
16. The computing device of claim 14 , further comprising introducing an additional bit-shift n adj to calculate ψ x (i, j), ψ y (i, j), and θ(i, j), wherein:
ψ
x
(
i
,
j
)
=
(
∂
I
(
1
)
∂
x
(
i
,
j
)
+
∂
I
(
0
)
∂
x
(
i
,
j
)
)
⪢
(
3
-
n
adj
)
;
ψ
y
(
i
,
j
)
=
(
∂
I
(
1
)
∂
y
(
i
,
j
)
+
∂
I
(
0
)
∂
y
(
i
,
j
)
)
⪢
(
3
-
n
adj
)
;
and
θ
(
i
,
j
)
=
(
I
(
1
)
(
i
,
j
)
⪢
(
6
-
n
adj
)
)
-
(
I
(
0
)
(
i
,
j
)
⪢
(
6
-
n
a
d
j
)
)
.
17. The computing device of claim 16 , wherein a value of n adj is set equal to 2.
18. The computing device of claim 16 , wherein the value of S 2 is calculated based on ψ x (i, j)·ψ y (i, j) for the set of prediction samples, and wherein the value of S 6 is calculated based on θ(i, j)·ψ y (i, j) for the set of prediction samples.
19. The computing device of claim 12 , wherein calculating gradient value
∂
I
(
k
)
∂
x
(
i
,
j
)
,
k=0, 1 and gradient value
∂
I
(
k
)
∂
y
(
i
,
j
)
,
k=0, 1 further comprises calculating a horizontal gradient and a vertical gradient at each of a first prediction position of the first prediction L 0 and a second prediction position of the second prediction L 1 .
20. The computing device of claim 19 , wherein the horizontal gradient is calculated at each of the first prediction position and the second prediction position as:
∂
I
(
k
)
∂
x
(
i
i
j
)
=
(
I
(
k
)
(
i
+
1
,
j
)
-
I
(
k
)
(
i
-
1
,
j
)
)
⪢
(
4
+
n
grad
)
,
k
=
0
,
1
;
and the vertical gradient is calculated at each of the first prediction position and the second prediction position as:
∂
I
(
k
)
∂
y
(
i
,
j
)
=
(
I
(
k
)
(
i
,
j
+
1
)
-
I
(
k
)
(
i
,
j
-
1
)
)
⪢
(
4
+
n
grad
)
,
k
=
0
,
1
,
wherein n grad is an additional right shift.
21. A non-transitory computer readable storage medium storing a plurality of programs for execution by a computing device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, cause the computing device to perform acts comprising:
acquiring, for a current picture, a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture is displayed before the current picture and the second reference picture is displayed after the current picture;
determining, for a current block of the current picture, a first prediction L 0 corresponding to the first reference picture;
determining, for the current block, a second prediction L 1 corresponding to the second reference picture;
calculating a respective motion refinement (v x , v y ) of each subblock of the current block by minimizing a difference between the first prediction L 0 and the second prediction L 1 based on a motion refinement threshold and a plurality of internal parameters representing different summation characters of the first prediction L 0 and the second prediction L 1 ;
calculating first gradient values
∂
I
(
0
)
∂
x
(
i
,
j
)
and
∂
I
(
0
)
∂
y
(
i
,
j
)
for the first prediction L 0 and second gradient values
∂
I
(
1
)
∂
x
(
i
,
j
)
and
∂
I
(
k
)
∂
y
(
i
,
j
)
for the second prediction L 1 , wherein I (0) (i, j) is a prediction sample at sample location (i, j) of the first prediction L 0 , and I (1) (i, j) is a prediction sample at the sample location (i, j) of the second prediction L 1 ; and
calculating a final bi-prediction of the current block based on the first prediction L 0 , the second prediction L 1 , respective motion refinements (v x , v y ), the first gradient values
∂
I
(
0
)
∂
x
(
i
,
j
)
and
∂
I
(
0
)
∂
y
(
i
,
j
)
,
and the second gradient values
∂
I
(
1
)
∂
x
(
i
,
j
)
and
∂
I
(
1
)
∂
y
(
i
,
j
)
.