Block boundary prediction refinement with optical flow
Systems, methods, and instrumentalities are disclosed for sub-block/block refinement, including sub-block/block boundary refinement, such as block boundary prediction refinement with optical flow (BBPROF). A block comprising a current sub-block may be decoded based on a sample value for a first pixel that is obtained based on, for example, an MV for a current sub-block, an MV for a sub-block adjacent the current sub-block, and a sample value for a second pixel adjacent the first pixel. BBPROF may include determining spatial gradients at pixel(s)/sample location(s). An MV difference may be calculated between a current sub-block and one or more neighboring sub-blocks. An MV offset may be determined at pixel(s)/sample location(s) based on the MV difference. A sample value offset for the pixel in a current sub-block may be determined. The prediction for a reference picture list may be refined by adding the calculated sample value offset to the sub-block prediction.
1. An apparatus for video decoding, comprising one or more processors, wherein the one or more processors are configured to:
obtain a sample value for a first pixel based on a motion vector (MV) for a current sub-block, a MV for a sub-block that is adjacent to the current sub-block, and a sample value for a second pixel that is adjacent to the first pixel, wherein the first pixel is a boundary pixel of the current sub-block; and
decode a block comprising the current sub-block based on the obtained sample value for the first pixel.
2. The apparatus of claim 1 , wherein the block comprises the first pixel, the second pixel, and a third pixel that is adjacent to the first pixel, wherein the one or more processors are configured to:
determine that the first pixel is the boundary pixel of the current sub-block; and
based on a determination that the first pixel is the boundary pixel of the current sub-block,
determine a difference between the MV for the current sub-block and the MV for the sub-block that is adjacent to the current sub-block,
determine a gradient for the first pixel based on the sample value for the second pixel and a sample value for the third pixel, and
determine a sample value offset based on the determined gradient and the difference between the MV for the current sub-block and the MV for the sub-block that is adjacent to the current sub-block, wherein the sample value for the first pixel is obtained based on the determined sample value offset.
3. The apparatus of claim 1 , wherein the one or more processors are further configured to determine a gradient based on at least the sample value for the second pixel, wherein the sample value for the first pixel is obtained using the gradient.
4. The apparatus of claim 1 , wherein obtaining the sample value for the first pixel based on the MV for the current sub-block, the MV for the sub-block that is adjacent to the current sub-block, and the sample value for the second pixel that is adjacent to the first pixel comprises obtaining the sample value for the first pixel based on the sample value for the second pixel that is adjacent to the first pixel and based on a MV difference between the MV for the current sub-block and the MV for the sub-block that is adjacent to the current sub-block.
5. The apparatus of claim 1 , wherein the sub-block that is adjacent to the current sub-block is a first sub-block, and the block comprises the first sub-block and a second sub-block that is adjacent to the current sub-block, wherein the sample value for the first pixel is obtained further based on a MV for the second sub-block.
6. The apparatus of claim 1 , wherein the first pixel and the second pixel are in the current sub-block.
7. The apparatus of claim 1 , wherein a weighting factor that varies in accordance with a distance of the first pixel from a corresponding boundary of the current sub-block is used to obtain the sample value for the first pixel.
8. The apparatus of claim 1 , further comprising:
at least one of (i) an antenna configured to receive a signal, the signal including data representative of an image, (ii) a band limiter configured to limit the received signal to a band of frequencies that includes the data representative of the image, or (iii) a display configured to display the image.
9. The apparatus of claim 1 , wherein the one or more processors are further configured to obtain a MV difference between the MV for the current sub-block and the MV for the sub-block that is adjacent to the current sub-block, wherein the sample value for the first pixel is obtained based on the MV difference.
10. An apparatus for video encoding, comprising one or more processors, wherein the one or more processors are configured to:
obtain a sample value for a first pixel based on a motion vector (MV) for a current sub-block, a MV for a sub-block that is adjacent to the current sub-block, and a sample value for a second pixel that is adjacent to the first pixel, wherein the first pixel is a boundary pixel of the current sub-block; and
encode a block comprising the current sub-block based on the obtained sample value for the first pixel.
11. The apparatus of claim 10 , wherein the block comprises the first pixel, the second pixel, and a third pixel that is adjacent to the first pixel, wherein the one or more processors are configured to:
determine that the first pixel is the boundary pixel of the current sub-block; and
based on a determination that the first pixel is the boundary pixel of the current sub-block,
determine a difference between the MV for the current sub-block and the MV for the sub-block that is adjacent to the current sub-block,
determine a gradient for the first pixel based on the sample value for the second pixel and a sample value for the third pixel, and
determine a sample value offset based on the determined gradient and the difference between the MV for the current sub-block and the MV for the sub-block that is adjacent to the current sub-block, wherein the sample value for the first pixel is obtained based on the determined sample value offset.
12. The apparatus of claim 10 , wherein obtaining the sample value for the first pixel based on the MV for the current sub-block, the MV for the sub-block that is adjacent to the current sub-block, and the sample value for the second pixel that is adjacent to the first pixel comprises obtaining the sample value for the first pixel based on the sample value for the second pixel that is adjacent to the first pixel and based on a MV difference between the MV for the current sub-block and the MV for the sub-block that is adjacent to the current sub-block.
13. The apparatus of claim 10 , wherein the one or more processors are further configured to determine a gradient for an optical flow model based on at least the sample value for the second pixel, and wherein the sample value for the first pixel is obtained using the optical flow model.
14. A method for video decoding, comprising:
obtaining a sample value for a first pixel based on a motion vector (MV) for a current sub-block, a MV for a sub-block that is adjacent to the current sub-block, and a sample value for a second pixel that is adjacent to the first pixel, wherein the first pixel is a boundary pixel of the current sub-block; and
decoding a block comprising the current sub-block based on the obtained sample value for the first pixel.
15. The method of claim 14 , wherein the block comprises the first pixel, the second pixel, and a third pixel that is adjacent to the first pixel, wherein obtaining the sample value for the first pixel comprises:
determining that the first pixel is the boundary pixel of the current sub-block;
based on a determination that the first pixel is the boundary pixel of the current sub-block,
determining a difference between the MV for the current sub-block and the MV for the sub-block that is adjacent to the current sub-block,
determining a gradient for the first pixel based on the sample value for the second pixel and a sample value for the third pixel, and
determining a sample value offset based on the determined gradient and the difference between the MV for the current sub-block and the MV for the sub-block that is adjacent to the current sub-block; and
obtaining the sample value for the first pixel based on the determined sample value offset.
16. The method of claim 14 , wherein obtaining the sample value for the first pixel based on the MV for the current sub-block, the MV for the sub-block that is adjacent to the current sub-block, and the sample value for the second pixel that is adjacent to the first pixel comprises obtaining the sample value for the first pixel based on the sample value for the second pixel that is adjacent to the first pixel and based on a MV difference between the MV for the current sub-block and the MV for the sub-block that is adjacent to the current sub-block.
17. The method of claim 14 , wherein a gradient for an optical flow model is determined based on at least the sample value for the second pixel and is used in the optical flow model to obtain the sample value for the first pixel.
18. A computer readable medium including instructions for causing one or more processors to perform the method of claim 14 .
19. A method for video encoding, comprising:
obtaining a sample value for a first pixel based on a motion vector (MV) for a current sub-block, a MV for a sub-block that is adjacent to the current sub-block, and a sample value for a second pixel that is adjacent to the first pixel, wherein the first pixel is a boundary pixel of the current sub-block; and
encoding a block comprising the current sub-block based on the obtained sample value for the first pixel.
20. The method of claim 19 , wherein the block comprises the first pixel, the second pixel, and a third pixel that is adjacent to the first pixel, and the method comprises:
determining that the first pixel is the boundary pixel of the current sub-block; and
based on a determination that the first pixel is the boundary pixel of the current sub-block,
determining a difference between the MV for the current sub-block and the MV for the sub-block that is adjacent to the current sub-block,
determining a gradient for the first pixel based on the sample value for the second pixel and a sample value for the third pixel, and
determining a sample value offset based on the determined gradient and the difference between the MV for the current sub-block and the MV for the sub-block that is adjacent to the current sub-block, wherein the sample value for the first pixel is obtained based on the determined sample value offset.