Inter prediction with refinement in video processing
Devices, systems and methods for digital video coding, which includes inter prediction with refinement, are described. An exemplary method of video processing includes determining to use, for a conversion between a current block of a video and a bitstream representation of the video, a first linear optimization model for the conversion using a first coding mode, the first linear optimization model being derived from a second linear optimization model that is used for the conversion using a second coding mode, and performing, based on the determining, the conversion. Another exemplary method of video processing includes determining to use, for a conversion between a current block of a video and a bitstream representation of the video, a gradient value computation algorithm for a bi-directional optical flow tool, and performing, based on the determining, the conversion.
1. A method of processing video data, comprising:
determining two corresponding regions for a sub-block of a current video block of a current picture of a video, wherein the two corresponding regions are in two reference pictures of the current video block, respectively;
deriving a sum of absolute difference (SAD) based on partial samples of the two corresponding regions;
determining that a first prediction mode is applied to the sub-block based on the SAD, wherein, in the first prediction mode, prediction samples of the sub-block are refined based on at least gradient values in different directions; and
performing, based on the first prediction mode, a conversion between the sub-block and a bitstream of the video;
wherein, in the first prediction mode, τ 0 ×τ 1 >0, wherein τ 0 is a difference between a picture order count of the current picture and a picture order count of a first reference picture (R 0 ) of the two reference pictures (τ 0 =POC(current)−POC(R 0 )), and wherein τ 1 is a difference between a picture order count of a second reference picture (R 1 ) of the two reference pictures and the picture order count of the current picture (τ 1 =POC(R 1 )−POC(current)).
2. The method of claim 1 , further comprising:
refraining from enabling the first prediction mode in response to abs(abs(τ 0 )−abs(τ 1 )≥T 0 , wherein T 0 is an integer, and T 0 is larger than 0.
3. The method of claim 1 , further comprising:
refraining from enabling the first prediction mode in response to abs(τ 0 )≥abs(τ 1 )×T 0 or abs(τ 1 )≥abs(τ 0 )×T 0 , wherein T 0 is an integer.
4. The method of claim 3 , wherein T 0 =1.
5. The method of claim 2 , wherein T 0 is signaled in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a tile group header, a tile, a coding unit (CU) or a coding tree unit (CTU).
6. The method of claim 1 , wherein the conversion comprises decoding the current video block from the bitstream.
7. The method of claim 1 , wherein the conversion comprises encoding the current video block into the bitstream.
8. The method of claim 1 , wherein the SAD is larger than or equal to a first threshold.
9. The method of claim 8 , wherein samples in one row of every R rows in each of the two corresponding regions are used to derive the SAD, and R is an integer larger than 1.
10. The method of claim 1 , wherein the gradient values comprise spatial gradient values, and spatial gradient values in different directions are derived based on samples of the corresponding regions and an arithmetic shifting operation with a shift value S, and wherein S is a fixed integer that is not equal to 4.
11. The method of claim 10 , wherein the spatial gradient values are derived based on samples from a same reference picture and outer samples of each of the two corresponding regions.
12. An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:
determine two corresponding regions for a sub-block of a current video block of a current picture of a video, wherein the two corresponding regions are in two reference pictures of the current video block, respectively;
derive a sum of absolute difference (SAD) based on partial samples of the two corresponding regions;
determine that a first prediction mode is applied to the sub-block based on the SAD, wherein, in the first prediction mode, prediction samples of the sub-block are refined based on at least gradient values in different directions; and
perform, based on the first prediction mode, a conversion between the sub-block and a bitstream of the video;
wherein, in the first prediction mode, τ 0 ×τ 1 >0, wherein τ 0 is a difference between a picture order count of the current picture and a picture order count of a first reference picture (R 0 ) of the two reference pictures (τ 0 =POC(current)−POC(R 0 )), and wherein τ 1 is a difference between a picture order count of a second reference picture (R 1 ) of the two reference pictures and the picture order count of the current picture (τ 1 =POC(R 1 )−POC(current)).
13. The apparatus of claim 12 , wherein the processor is further caused to:
refrain from enabling the first prediction mode in response to abs(abs(τ 0 )−abs(τ 1 ))≥T 0 , wherein T 0 is an integer, and T 0 is larger than 0.
14. The apparatus of claim 12 , wherein the processor is further caused to:
refrain from enabling the first prediction mode in response to abs(τ 0 )≥abs(τ 1 )×T 0 or abs(τ 1 )≥abs(τ 0 )×T 0 , wherein T 0 is an integer.
15. The apparatus of claim 14 , wherein T 0 =1.
16. The apparatus of claim 13 , wherein T 0 is signaled in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a tile group header, a tile, a coding unit (CU) or a coding tree unit (CTU).
17. The apparatus of claim 12 , wherein the conversion comprises decoding the current video block from the bitstream.
18. The apparatus of claim 12 , wherein the conversion comprises encoding the current video block into the bitstream.
19. A non-transitory computer-readable storage medium storing instructions that cause a processor to:
determine two corresponding regions for a sub-block of a current video block of a current picture of a video, wherein the two corresponding regions are in two reference pictures of the current video block, respectively;
derive a sum of absolute difference (SAD) based on partial samples of the two corresponding regions;
determine that a first prediction mode is applied to the sub-block based on the SAD, wherein, in the first prediction mode, prediction samples of the sub-block are refined based on at least gradient values in different directions; and
perform, based on the first prediction mode, a conversion between the sub-block and a bitstream of the video;
wherein, in the first prediction mode, τ 0 ×τ 1 >0, wherein τ 0 is a difference between a picture order count of the current picture and a picture order count of a first reference picture (R 0 ) of the two reference pictures (τ 0 =POC(current)−POC(R 0 )), and wherein τ 1 is a difference between a picture order count of a second reference picture (R 1 ) of the two reference pictures and the picture order count of the current picture (τ 1 =POC(R 1 )−POC(current)).
20. A non-transitory computer-readable recording medium storing a bitstream of a video which is generated by a method performed by a video processing apparatus, the method comprising:
determining two corresponding regions for a sub-block of a current video block of a current picture of a video, wherein the two corresponding regions are in two reference pictures of the current video block, respectively;
deriving a sum of absolute difference (SAD) based on partial samples of the two corresponding regions;
determining that a first prediction mode is applied to the sub-block based on the SAD, wherein, in the first prediction mode, prediction samples of the sub-block are refined based on at least gradient values in different directions; and
generating, based on the first prediction mode, the bitstream;
wherein, in the first prediction mode, τ 0 ×τ 1 >0, wherein τ 0 is a difference between a picture order count of the current picture and a picture order count of a first reference picture (R 0 ) of the two reference pictures (τ 0 =POC(current)−POC(R 0 )), and wherein τ 1 is a difference between a picture order count of a second reference picture (R 1 ) of the two reference pictures and the picture order count of the current picture (τ 1 =POC(R 1 )−POC(current)).