Methods and apparatuses for prediction refinement with optical flow
Methods, apparatuses, and non-transitory computer-readable storage mediums are provided for encoding a video signal. A method includes obtaining a first reference picture associated with a video block in a current picture within the video signal and a first motion vector (MV) from the video block to a reference block; obtaining first prediction samples of the video block generated from samples of the reference block; controlling internal bit-depths of internal prediction refinement with optical flow (PROF) parameters by performing a shift operation; obtaining a prediction refinement value for the first prediction sample based on the horizontal and vertical gradient values and the horizontal and vertical motion differences; obtaining second prediction samples associated with a second MV and a prediction refinement value for a second prediction sample; and obtaining a final prediction sample of the video block based on the first prediction sample, the second prediction sample, and the prediction refinement values.
1. A method for coding a video signal, comprising:
obtaining a first reference picture associated with a video block in a current picture within the video signal and a first motion vector (MV) from the video block in the current picture to a reference block in the first reference picture, wherein the first reference picture comprises multiple non-overlapped video blocks and at least one video block is associated with at least one MV;
obtaining first prediction samples of the video block generated from samples of the reference block in the first reference picture;
controlling internal bit-depths of internal prediction refinement with optical flow (PROF) parameters by performing a shift operation according to shift values, wherein the shift values comprise a first shift value for calculation of gradient values that are used in a bi-directional optical flow (BDOF) process, and wherein the internal PROF parameters comprise a horizontal gradient value, a vertical gradient value, a horizontal motion difference, and a vertical motion difference derived for a first prediction sample;
obtaining a prediction refinement value for the first prediction sample based on the horizontal and vertical gradient values and the horizontal and vertical motion differences;
obtaining second prediction samples associated with a second MV and a corresponding prediction refinement value for a second prediction sample; and
obtaining a final prediction sample of the video block based on the first prediction sample, the second prediction sample, and the prediction refinement values.
2. The method of claim 1 , wherein controlling the internal bit-depths of the internal PROF parameters comprises, for one sample in the video block:
obtaining a horizontal gradient value based on prediction samples of two samples adjacent to said one sample in a horizontal direction and the first shift value; and
obtaining a vertical gradient value based on prediction samples of two samples adjacent to said one sample in a vertical direction and the first shift value.
3. The method of claim 2 , wherein the first shift value is equal to a maximum of 6 and a value of coding bit-depth minus 6.
4. The method of claim 2 , further comprising, for said one sample in the video block:
obtaining control point MVs of a coding block which contains the video block, wherein the control point MVs comprises MVs of top-left, top-right, and bottom-left corner blocks of the coding block;
obtaining affine model parameters derived based on the control point MVs;
obtaining a horizontal MV difference and a vertical MV difference based on the affine parameters, the horizontal offset, and the vertical offset;
right shifting the horizontal MV difference by a second shift value; and
right shifting the vertical MV difference by the second shift value.
5. The method of claim 4 , wherein the second shift value is equal to 13 minus a precision bit-depth of gradient values.
6. The method of claim 5 , wherein the precision bit-depth of gradient values is equal to a maximum of 6 and a value of coding bit-depth minus 6.
7. The method of claim 4 , wherein obtaining the final prediction sample of the video block comprises:
obtaining first prediction refinement values based on horizontal gradient values, horizontal MV differences, vertical gradient values, and vertical MV differences that are generated for the first prediction sample;
obtaining second prediction refinement values based on horizontal gradient values, horizontal motion differences, vertical gradient values, and vertical motion differences that are generated for the second prediction sample;
obtaining the prediction refinement values by averaging the first prediction refinement values and the second prediction refinement values;
obtaining a bi-prediction sample based on a sum of the first prediction sample, the second prediction sample, the prediction refinement values; and
right shifting the sum by a third shift value.
8. The method of claim 2 , wherein obtaining the horizontal and vertical gradient values of the first prediction sample further comprises:
deriving an additional row or column of prediction samples outside the video block which is adjacent to each of top, left, bottom and right boundaries of the video block for the first prediction sample.
9. The method of claim 8 , wherein deriving the additional row or column of the prediction samples further comprises:
deriving prediction samples outside the video block which are adjacent to the left and right boundaries from integer reference samples in the first reference picture left to a fractional sample position; and
deriving prediction samples outside the video block which are adjacent to the top and bottom boundaries from integer reference samples in the first reference picture above the fractional sample position.
10. The method of claim 8 , wherein deriving the additional row or column of the prediction samples further comprises:
deriving prediction samples outside the video block which are adjacent to the left and right boundaries from integer reference samples in the first reference picture that are closest to respective fractional sample positions in a horizontal direction; and
deriving prediction samples outside the video block which are adjacent to the top and bottom boundaries from integer reference samples in the first reference picture that are closest to the respective fractional sample positions in a vertical direction.
11. A computing device for coding a video signal, comprising:
one or more processors;
a non-transitory computer-readable storage medium storing instructions executable by the one or more processors, wherein the one or more processors are configured to:
obtain a first reference picture associated with a video block in a current picture within the video signal and a first motion vector (MV) from the video block in the current picture to a reference block in the first reference picture, wherein the first reference picture comprises multiple non-overlapped video blocks and at least one video block is associated with at least one MV;
obtain first prediction samples of the video block generated from samples of the reference block in the first reference picture;
control internal bit-depths of internal prediction refinement with optical flow (PROF) parameters by performing a shift operation according to shift values, wherein the shift values comprise a first shift value for calculation of gradient values that are used in a bi-directional optical flow (BDOF) process, and wherein the internal PROF parameters comprise a horizontal gradient value, a vertical gradient value, a horizontal motion difference, and a vertical motion difference derived for a first prediction sample;
obtain a prediction refinement value for the first prediction sample based on the horizontal and vertical gradient values and the horizontal and vertical motion differences;
obtain second prediction samples associated with a second MV and a corresponding prediction refinement value for a second prediction sample; and
obtain a final prediction sample of the video block based on the first prediction sample, the second prediction sample, and the prediction refinement values.
12. The computing device of claim 11 , wherein the one or more processors configured to control the internal bit-depths of the internal PROF parameters are further configured to, for one sample in the video block:
obtain a horizontal gradient value based on prediction samples of two samples adjacent to said one sample in a horizontal direction and the first shift value; and
obtain a vertical gradient value based on prediction samples of two samples adjacent to said one sample in a vertical direction and the first shift value.
13. The computing device of claim 12 , wherein the first shift value is equal to a maximum of 6 and a value of coding bit-depth minus 6.
14. The computing device of claim 12 , wherein the one or more processors are further configured to, for said one sample in the video block:
obtain control point MVs of a coding block which contains the video block, wherein the control point MVs comprises MVs of top-left, top-right, and bottom-left corner blocks of the coding block;
obtain affine model parameters derived based on the control point MVs;
obtain a horizontal MV difference and a vertical MV difference based on the affine parameters, the horizontal offset, and the vertical offset;
right shift the horizontal MV difference by a second shift value; and
right shift the vertical MV difference by the second shift value.
15. The computing device of claim 14 , wherein the second shift value is equal to 13 minus a precision bit-depth of gradient values.
16. The computing device of claim 14 , wherein the one or more processors configured to obtain the final prediction sample of the video block are further configured to:
obtain first prediction refinement values based on horizontal gradient values, horizontal MV differences, vertical gradient values, and vertical MV differences that are generated for the first prediction sample;
obtain second prediction refinement values based on horizontal gradient values, horizontal motion differences, vertical gradient values, and vertical motion differences that are generated for the second prediction sample;
obtain the prediction refinement values by averaging the first prediction refinement values and the second prediction refinement values;
obtain a bi-prediction sample based on a sum of the first prediction sample, the second prediction sample, the prediction refinement values; and
right shift the sum by a third shift value.
17. The computing device of claim 12 , wherein the one or more processors configured to obtain the horizontal and vertical gradient values of the first prediction sample are further configured to:
derive an additional row or column of prediction samples outside the video block which is adjacent to each of top, left, bottom and right boundaries of the video block for the first prediction sample.
18. The computing device of claim 17 , wherein the one or more processors configured to derive the additional row or column of the prediction samples are further configured to:
derive prediction samples outside the video block which are adjacent to the left and right boundaries from integer reference samples in the first reference picture left to a fractional sample position; and
derive prediction samples outside the video block which are adjacent to the top and bottom boundaries from integer reference samples in the first reference picture above the fractional sample position.
19. The computing device of claim 17 , wherein the one or more processors configured to derive the additional row or column of the prediction samples are further configured to:
derive prediction samples outside the video block which are adjacent to the left and right boundaries from integer reference samples in the first reference picture that are closest to respective fractional sample positions in a horizontal direction; and
derive prediction samples outside the video block which are adjacent to the top and bottom boundaries from integer reference samples in the first reference picture that are closest to the respective fractional sample positions in a vertical direction.
20. A non-transitory computer readable storage medium comprising a bitstream to be decoded by a method for coding a video signal comprising:
obtaining a first reference picture associated with a video block in a current picture within the video signal and a first motion vector (MV) from the video block in the current picture to a reference block in the first reference picture, wherein the first reference picture comprises multiple non-overlapped video blocks and at least one video block is associated with at least one MV;
obtaining first prediction samples of the video block generated from samples of the reference block in the first reference picture;
controlling internal bit-depths of internal prediction refinement with optical flow (PROF) parameters by performing a shift operation according to shift values, wherein the shift values comprise a first shift value for calculation of gradient values that are used in a bi-directional optical flow (BDOF) process, and wherein the internal PROF parameters comprise a horizontal gradient value, a vertical gradient value, a horizontal motion difference, and a vertical motion difference derived for a first prediction sample;
obtaining a prediction refinement value for the first prediction sample based on the horizontal and vertical gradient values and the horizontal and vertical motion differences;
obtaining second prediction samples associated with a second MV and a corresponding prediction refinement value for a second prediction sample; and
obtaining a final prediction sample of the video block based on the first prediction sample, the second prediction sample, and the prediction refinement values.