Gradient computation in bi-directional optical flow
Gradient computation in Bi-directional optical flow are described. In a representative aspect, a method of video processing including: modifying temporal gradients in a motion vectors refinement process of a video block or sub-blocks within the video block; and performing the motion vectors refinement process of the video block or the sub-blocks within the video block based on the modified temporal gradients.
1. A method of video processing, comprising:
performing, in a motion vector refinement process, for a temporal gradient of a first block or sub-block with a first size, a temporal gradient modification process to generate modified temporal gradients of the first block or sub-block; and
performing the motion vector refinement process for a second block or sub-block with a second size according to the modified temporal gradients of the first block or sub-block, wherein the first size is different from the second size;
wherein, in the temporal gradient modification process, a temporal gradient is modified by removing a mean difference between a reference block 0 and a reference block 1 for the first block or sub-block, and
wherein the first block or sub-block comprises M 1 ×N 1 samples, and the second block or sub-block comprises M 2 ×N 2 samples, where M 1 , N 1 , M 2 and N 2 are integers, and where M 1 =N 1 =8 or M 1 =N 1 =4, and where M 2 =N 2 =8 or M 2 =N 2 =4, and deriving refined motion vectors for a third block or sub-blocks within the third block by using partial temporal and/or spatial gradients that are calculated using partial pixels of the third block or sub-blocks within the third block such that pixels from every N rows and/or columns are used to calculate the temporal and/or spatial gradients, where N is an integer.
2. The method of claim 1 , wherein the motion vector refinement process further comprises:
deriving, refined motion vectors of the second block or sub-block, using the modified temporal gradients.
3. The method of claim 1 , wherein the motion vector refinement process is implemented in a bi-directional optical flow (BIO) technique or in a decoder-side motion vector refinement (DMVR) technique or a frame-rate up conversion (FRUC) technique.
4. The method of claim 1 , wherein N=2.
5. The method of claim 1 , wherein pixels from top-left/top-right/bottom-left/bottom-right quarter of the third block or the sub-blocks within the third block are used to calculate the temporal and/or the spatial gradients.
6. The method of claim 1 , wherein the refined motion vectors for the third block or sub-blocks within the third block are derived during a two-step motion vector refinement process or a multiple-step motion vector refinement process, wherein the temporal and/or the spatial gradients are only used to derive refined motion vectors of the third block or the sub-blocks within the third block.
7. The method of claim 1 , wherein the third block corresponds to a prediction unit or a coding unit.
8. The method of claim 1 , further comprising performing a conversion between the first block, the second block and the third block and a bitstream of the first block, the second block and the third block.
9. The method of claim 8 , wherein the conversion generates the first block, the second block and the third block from the bitstream.
10. The method of claim 8 , wherein the conversion generates the bitstream from the first block, the second block, and the third block.
11. A video processing apparatus comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to implement:
perform, in a motion vector refinement process, for a temporal gradient of a first block or sub-block with a first size, a temporal gradient modification process to generate modified temporal gradients of the first block or sub-block; and
perform the motion vectors vector refinement process for a second block or sub-block with a second size according to the modified temporal gradients of the first block or sub-block, wherein the first size is different from the second size;
wherein, in the temporal gradient modification process, a temporal gradient is modified by removing a mean difference between a reference block 0 and a reference block 1 for the first block or sub-block is removed, and
wherein the first block or sub-block comprises M 1 ×N 1 samples, and the second block or sub-block comprises M 2 ×N 2 samples, where M 1 , N 1 , M 2 and N 2 are integers, and where M 1 =N 1 =8 or M 1 =N 1 =4, and where M 2 =N 2 =8 or M 2 =N 2 =4, and
derive refined motion vectors for a third block or sub-blocks within the third block by using partial temporal and/or spatial gradients that are calculated using partial pixels of the third block or sub-blocks within the third block such that pixels from every N rows and/or columns are used to calculate the temporal and/or spatial gradients, where N is an integer.
12. The video processing apparatus of claim 11 , wherein the motion vector refinement process further comprises:
derive, refined motion vectors of the second block or sub-block, using the modified temporal gradients.
13. A computer program product stored on a non-transitory computer readable media, the computer program product including program code for carrying out a method, comprising: performing, in a motion vector refinement process, for a temporal gradient of a first block or sub-block with a first size, a temporal gradient modification process to generate modified temporal gradients of the first block or sub-block; and
performing the motion vector refinement process for a second block or sub-block with a second size according to the modified temporal gradients of the first block or sub-block, wherein the first size is different from the second size;
wherein, in the temporal gradient modification process, a temporal gradient is modified by removing a mean difference between a reference block 0 and a reference block 1 for the first block or sub-block, and
wherein the first block or sub-block comprises M 1 ×N 1 samples, and the second block or sub-block comprises M 2 ×N 2 samples, where M 1 , N 1 , M 2 and N 2 are integers, and where M 1 =N 1 =8 or M 1 =N 1 =4, and where M 2 =N 2 =8 or M 2 =N 2 =4, and
deriving refined motion vectors for a third block or sub-blocks within the third block by using partial temporal and/or spatial gradients that are calculated using partial pixels of the third block or sub-blocks within the third block such that pixels from every N rows and/or columns are used to calculate the temporal and/or spatial gradients, where N is an integer.
14. The video processing apparatus of claim 11 , wherein the motion vector refinement process is implemented in a bi-directional optical flow (BIO) technique or in a decoder-side motion vector refinement (DMVR) technique or a frame-rate up conversion (FRUC) technique.
15. The video processing apparatus of claim 11 , wherein N=2.