Complexity reduction and bit-width control for bi-directional optical flow
Systems and methods are described for reducing the complexity of using bi-directional optical flow (BIO) in video coding. In some embodiments, bit-width reduction steps are introduced in the BIO motion refinement process to reduce the maximum bit-width used for BIO calculations. In some embodiments, simplified interpolation filters are used to generate predicted samples in an extended region around a current coding unit. In some embodiments, different interpolation filters are used for vertical versus horizontal interpolation. In some embodiments, BIO is disabled for coding units with small heights and/or for coding units that are predicted using a sub-block level inter prediction technique, such as advanced temporal motion vector prediction (ATMVP) or affine prediction.
1. A method comprising:
obtaining a first array of first-component gradients based on a first prediction signal array I (0) (i, j) from a first reference picture, where (i, j) are coordinates of samples;
obtaining a second array of first-component gradients based on a second prediction signal array I (1) (i, j) from a second reference picture;
calculating a reduced-bit-width first-component intermediate parameter array including performing a right bit shift on a sum of (i) the first array of first-component gradients and (ii) the second array of first-component gradients;
calculating at least a first-component motion refinement based at least in part on the reduced-bit-width first-component intermediate parameter array; and
generating a prediction of a current block in a video with bi-directional optical flow using at least the first-component motion refinement.
2. The method of claim 1 , further comprising:
calculating a signal-difference parameter array θ(i, j) by a method comprising calculating a difference between the first prediction signal array I (0) (i, j) and the second prediction signal array I (1) (i, j); and
calculating a signal-first-component-gradient correlation parameter by summing components of an elementwise multiplication of (i) the signal-difference parameter array θ(i, j) with (ii) the first-component intermediate parameter array;
wherein calculating the first-component motion refinement v x comprises bit-shifting the signal-first-component-gradient correlation parameter to obtain the first-component motion refinement.
3. The method of claim 2 , wherein calculating the signal-difference parameter array θ(i, j) comprises performing a right bit shift on each of the first prediction signal array I (0) (i, j) and the second prediction signal array I (1) (i, j) before calculating the difference between the first prediction signal array I (0) (i, j) and the second prediction signal array I (1) (i, j).
4. The method of claim 1 , wherein the first component is a horizontal component, with the first array of first-component gradients being a first horizontal gradient array
∂
I
(
0
)
∂
x
(
i
,
j
)
,
where
∂
I
(
0
)
∂
x
(
i
,
j
)
is an array of horizontal gradients of the first prediction signal array I (0) (i, j) at respective coordinates (i, j) the second array of first-component gradients being a second horizontal gradient array
∂
I
(
1
)
∂
x
(
i
,
j
)
,
where
∂
I
(
1
)
∂
x
(
i
,
j
)
is an array of horizontal gradients of the second prediction signal array I (1) (i, j) at respective coordinates (i, j), the reduced-bit-width first-component intermediate parameter array being a reduced-bit-width horizontal intermediate parameter array ψ x (i, j) calculated by a method comprising performing a right bit shift on a sum of (i) the first horizontal gradient array
∂
I
(
0
)
∂
x
(
i
,
j
)
and (ii) the second horizontal gradient array
∂
I
(
1
)
∂
x
(
i
,
j
)
,
and the first-component motion refinement being a horizontal motion refinement v x , the method further comprising:
calculating a first vertical gradient array
∂
I
(
0
)
∂
y
(
i
,
j
)
based on a first prediction signal array I (0) (i, j) from a first reference picture, where
∂
I
(
0
)
∂
y
(
i
,
j
)
is an array of vertical gradients of the first prediction signal array I (0) (i, j) at respective coordinates (i, j);
calculating a second vertical gradient array
∂
I
(
0
)
∂
y
(
i
,
j
)
based on a first prediction signal array I (0) (i, j) from a first reference picture, where
∂
I
(
1
)
∂
y
(
i
,
j
)
is an array of vertical gradients of the first prediction signal array I (1) (i, j) at respective coordinatse (i, j);
calculating a reduced-bit-width vertical intermediate parameter array ψ y (i, j) by a method comprising performing a right bit shift on a sum of (i) the first vertical gradient array and (ii) the second vertical gradient array; and
calculating a vertical motion refinement v y based at least in part on the reduced-bit-width horizontal intermediate parameter array ψ x (i, j) and the reduced-bit-width vertical intermediate parameter array ψ y (i, j);
wherein the prediction of the current block is generated using the horizontal motion refinement v x and the vertical motion refinement v y .
5. The method of claim 4 , further comprising:
calculating a cross-gradient correlation parameter S 2 by a method comprising summing components of an elementwise multiplication of (i) the horizontal intermediate parameter array ψ x (i, j) with (ii) the vertical intermediate parameter array ψ y (i, j);
wherein calculating the vertical motion refinement v y comprises determining a product of (i) the horizontal motion refinement v x and (ii) the cross-gradient correlation parameter S 2 .
6. The method of claim 5 , wherein determining the product of (i) the horizontal motion refinement v x and (ii) the cross-gradient correlation parameter S 2 comprises:
separating the cross-gradient correlation parameter S 2 into a most-significant-bit, MSB, parameter portion S 2,m and a least-significant-bit, LSB, parameter portion S 2,s ;
determining an MSB product of (i) the horizontal motion refinement v x and (ii) the MSB parameter portion S 2,m ;
determining an LSB product of (i) the horizontal motion refinement v x and (ii) the LSB parameter portion S 2,S ;
performing a left bit shift of the MSB product to generate a bit-shifted MSB product; and
adding the LSB product and the bit-shifted MSB product.
7. The method of claim 4 , wherein generating a prediction of the current block with bi-directional optical flow comprises:
for each sample in the current block, calculating a bi-directional-optical-flow sample offset, b, based on: (i) the first horizontal gradient array,
∂
I
(
0
)
∂
x
(
i
,
j
)
,
(ii) the first vertical gradient array
∂
I
(
0
)
∂
y
(
i
,
j
)
,
(iii) the second horizontal gradient array,
∂
I
(
1
)
∂
x
(
i
,
j
)
,
(iv) the second vertical gradient array,
∂
I
(
1
)
∂
y
(
i
,
j
)
,
(v) the horizontal motion refinement, v x , and (vi) the vertical motion refinement, v y ; and
for each sample in the current block, calculating a sum of at least the first prediction signal array, I (0) (i, j), the second prediction signal array, I (1) (i, j), and the bi-directional-optical-flow sample offset, b.
8. The method of claim 4 , wherein calculating each of the gradient arrays
∂
I
(
0
)
∂
x
(
i
,
j
)
,
∂
I
(
0
)
∂
y
(
i
,
j
)
,
∂
I
(
1
)
∂
x
(
i
,
j
)
,
∂
I
(
1
)
∂
y
(
i
,
j
)
comprises padding samples outside the prediction signal arrays I (0) (i, j), I (1) (i, j), with respective nearest boundary samples inside the prediction signal arrays.
9. The method of claim 2 , wherein calculating at least some values of the signal-difference parameter array θ(i, j) comprises padding samples outside the prediction signal arrays I (0) (i, j), I (1) (i, j), with respective nearest boundary samples inside the prediction signal arrays.
10. The method of claim 1 , wherein calculating at least some values of the first-component intermediate parameter array comprises padding gradient values outside the first and second arrays of first-component gradients with respective nearest boundary samples inside the first and second arrays of first-component gradients.
11. The method of claim 4 , wherein calculating at least some values of the vertical intermediate parameter array ψ y (i, j) comprises padding gradient values outside the vertical gradient arrays
∂
I
(
0
)
∂
y
(
i
,
j
)
,
∂
I
(
1
)
∂
y
(
i
,
j
)
with respective nearest boundary samples inside the vertical gradient arrays.
12. The method of claim 1 , performed by an encoder.
13. The method of claim 1 , performed by a decoder.
14. An apparatus comprising a processor configured to perform at least:
obtaining a first array of first-component gradients based on a first prediction signal array I (0) (i, j) from a first reference picture, where (i, j) are coordinates of samples;
obtaining a second array of first-component gradients based on a second prediction signal array I (1) (i, j) from a second reference picture;
calculating a reduced-bit-width first-component intermediate parameter array including performing a right bit shift on a sum of (i) the first array of first-component gradients and (ii) the second array of first-component gradients;
calculating at least a first-component motion refinement based at least in part on the reduced-bit-width first-component intermediate parameter array; and
generating a prediction of a current block in a video with bi-directional optical flow using at least the first-component motion refinement.
15. The apparatus of claim 14 , wherein the processor is further configured to perform:
calculating a signal-difference parameter array θ(i, j) by a method comprising calculating a difference between the first prediction signal array I (0) (i, j) and the second prediction signal array I (1) (i, j); and
calculating a signal-first-component-gradient correlation parameter by summing components of an elementwise multiplication of (i) the signal-difference parameter array θ(i, j) with (ii) the first-component intermediate parameter array;
wherein calculating the first-component motion refinement v x , comprises bit-shifting the signal-first-component-gradient correlation parameter to obtain the first-component motion refinement.
16. The apparatus of claim 15 , wherein calculating the signal-difference parameter array θ(i, j) comprises performing a right bit shift on each of the first prediction signal array I (0) (i, j) and the second prediction signal array I (1) (i, j) before calculating the difference between the first prediction signal array I (0) (i, j) and the second prediction signal array I (1) (i, j).
17. The apparatus of claim 14 , wherein the first component is a horizontal component, with the first array of first-component gradients being a first horizontal gradient array
∂
I
(
0
)
∂
x
(
i
,
j
)
,
where
∂
I
(
0
)
∂
x
(
i
,
j
)
is an array of horizontal gradients of the first prediction signal array I (0) (i, j) at respective coordinates (i, j), the second array of first-component gradients being a second horizontal gradient array
∂
I
(
1
)
∂
x
(
i
,
j
)
,
where
∂
I
(
1
)
∂
x
(
i
,
j
)
is an array of horizontal gradients of the second prediction signal array I (1) (i, j) at respective coordinates (i, j), the reduced-bit-width first-component intermediate parameter array being a reduced-bit-width horizontal intermediate parameter array ψ x (i, j) calculated by a method comprising performing a right bit shift on a sum of (i) the first horizontal gradient array
∂
I
(
0
)
∂
x
(
i
,
j
)
and (ii) the second horizontal gradient array
∂
I
(
1
)
∂
x
(
i
,
j
)
,
and the first-component motion refinement being a horizontal motion refinement v x , the processor further being configured to perform:
calculating a first vertical gradient array
∂
I
(
0
)
∂
y
(
i
,
j
)
based on a first prediction signal array I (0) (i, j) from a first reference picture, where
∂
I
(
0
)
∂
y
(
i
,
j
)
is an array of vertical gradients of the first prediction signal array I (0) (i, j) at respective coordinates (i, j);
calculating a second vertical gradient array
∂
I
(
1
)
∂
y
(
i
,
j
)
based on a second prediction signal array I (1) (i, j) from a second reference picture, where
∂
I
(
1
)
∂
y
(
i
,
j
)
is an array of vertical gradients of the first prediction signal array I (1) (i, j) at respective coordinates (i, j);
calculating a reduced-bit-width vertical intermediate parameter array ψ y (i, j) by a method comprising performing a right bit shift on a sum of (i) the first vertical gradient array and (ii) the second vertical gradient array; and
calculating a vertical motion refinement v y based at least in part on the reduced-bit- width horizontal intermediate parameter array ω x (i, j) and the reduced-bit-width vertical intermediate parameter array ψ y (i, j);
wherein the prediction of the current block is generated using the horizontal motion refinement v x and the vertical motion refinement v y .
18. The apparatus of claim 17 , wherein the processor is further configured to perform:
calculating a cross-gradient correlation parameter S 2 by a method comprising summing components of an elementwise multiplication of (i) the horizontal intermediate parameter array ψ x (i, j) with (ii) the vertical intermediate parameter array ψ y (i, j);
wherein calculating the vertical motion refinement v y comprises determining a product of (i) the horizontal motion refinement v x and (ii) the cross-gradient correlation parameter S 2 .
19. The apparatus of claim 18 , wherein determining the product of (i) the horizontal motion refinement v x and (ii) the cross-gradient correlation parameter S 2 comprises:
separating the cross-gradient correlation parameter S 2 into a most-significant-bit, MSB, parameter portion S 2,m and a least-significant-bit, LSB, parameter portion S 2,s ;
determining an MSB product of (i) the horizontal motion refinement v x and (ii) the MSB parameter portion S 2,m ;
determining an LSB product of (i) the horizontal motion refinement v x and (ii) the LSB parameter portion S 2,S ;
performing a left bit shift of the MSB product to generate a bit-shifted MSB product; and
adding the LSB product and the bit-shifted MSB product.
20. The apparatus of claim 17 , wherein generating a prediction of the current block with bi-directional optical flow comprises:
for each sample in the current block, calculating a bi-directional-optical-flow sample offset, b, based on: (i) the first horizontal gradient array,
∂
I
(
0
)
∂
x
(
i
,
j
)
,
(ii) the first vertical gradient array
∂
I
(
0
)
∂
y
(
i
,
j
)
,
(iii) the second horizontal gradient array,
∂
I
(
1
)
∂
x
(
i
,
j
)
,
(iv) the second vertical gradient array,
∂
I
(
1
)
∂
y
(
i
,
j
)
,
(v) the horizontal motion refinement, v x , and (vi) the vertical motion refinement, v y ; and
for each sample in the current block, calculating a sum of at least the first prediction signal array, I (0) (i, j), the second prediction signal array, I (1) (i, j), and the bi-directional-optical-flow sample offset, b.
21. The apparatus of claim 14 , wherein the apparatus is a video encoder.
22. The apparatus of claim 14 , wherein the apparatus is a video decoder.