Granted Patent B2

US 12,634,477 · App. 18/502,646 · Granted May 19, 2026

Inventors: Kai Zhang (San Diego, CA); Li Zhang (San Diego, CA); Hongbin Liu (Beijing, CN); Yue Wang (Beijing, CN)

Assignees: BEIJING BYTEDANCE NETWORK TECHNOLOGY CO., LTD.; BYTEDANCE INC.

H04N19/159H04N19/105H04N19/117H04N19/132H04N19/172H04N19/176H04N19/186H04N19/513H04N19/96

Abstract

Claims (64)

1 . A method of processing video data, comprising:

determining, for a conversion between a current block of a video and a bitstream of the video, two corresponding regions for a sub-block of the current block, wherein the two corresponding regions are in two reference pictures of the current block respectively;

deriving a sum of absolute difference (SAD) based on partial samples of the two corresponding regions, wherein positions of the partial samples of the two corresponding regions are predetermined;

determining that a first prediction mode is applied to the sub-block when the SAD is larger than or equal to a first threshold, wherein in the first prediction mode, prediction samples of the current block is refined at least based on spatial gradient values in different directions, wherein the first prediction mode is a bi-directional optical flow (BDOF) prediction mode; and

deriving the spatial gradient values in different directions based on samples of the two corresponding regions and an arithmetic shifting operation with a shift value S which is a fixed integer and is not equal to 4, wherein the spatial gradient values are derived based on samples from a same reference picture,

wherein whether the first prediction mode being applied to the sub-block is further determined based on one or more picture order count (POC) parameters associated with a current picture comprising the current block of the video,

wherein a first reference picture (R 0 ) and a second reference picture (R 1 ) of the two reference pictures are associated with the current picture, wherein τ 0 is a difference between a picture order count of the current picture and a picture order count of the first reference picture (τ 0 =POC(current)−POC(R 0 )), and wherein τ 1 is a difference between a picture order count of the second reference picture and the picture order count of the current picture (τ 1 =POC(R 1 )−POC(current)),

wherein the first prediction mode is disabled in response to abs (abs(τ 0 )−abs(τ 1 ))≥T 0 , wherein T 0 is an integer, and T 0 is larger than 0; or wherein the first prediction mode is disabled in response to abs(τ 0 )≥abs(τ 1 )×T 0 or abs(τ 1 )≥abs(τ 0 )×τ 0 , wherein T 0 is an integer, wherein T 0 =1, and

wherein in the first prediction mode, before deriving the spatial gradient values, one or more outer samples of the current block are derived with a padding operation instead of a first interpolation filtering operation, and a second interpolation filtering operation is performed for the current block;

wherein a size of the current block is M1×N1, wherein a first number of samples required by a gradient calculation is (M1+G1)× (N1+G1), wherein an interpolation filter used in the second interpolation filtering operation for a luma component of the current block comprises L1 taps, wherein a second number of samples required by the second interpolation filtering operation without enabling the first prediction mode is (M1+G1+L1−1)× (N1+G1+L1−1), wherein a third number of samples required by the second interpolation filtering operation with enabling the first prediction mode is (M1+k1+L1−1)× (N1+k1+L1−1), wherein M1, N1, G1, and L1 are positive integers, and wherein k1 is an integer less than G1.

2 . The method of claim 1 , wherein the spatial gradient values are derived further based on outer samples of each of the two corresponding regions,

wherein a number of samples used to derive a corresponding region and the outer samples is (M2+k2+L2−1)× (N2+k2+L2−1),

wherein M2 is a width of the corresponding region, N2 is a height of the corresponding region, L2 is a tap number of an interpolation filter, and k2 is an integer less than G2 which is a number of outer sample lines or columns, and

wherein L2=8, G2=2 and M2=N2=16.

3 . The method of claim 1 , wherein S is equal to 6, or

wherein the shift value S is independent of a sample bit-depth of the current block.

4 . The method of claim 1 , wherein samples in one row of every R rows in each of the two corresponding regions are used to derive the SAD, and R is an integer larger than 1.

5 . The method of claim 1 , wherein the method further comprises:

deriving a horizontal motion offset and a vertical motion offset based on the spatial gradient values;

deriving a prediction refinement based on the horizontal motion offset, the vertical motion offset, and the spatial gradient values;

deriving final prediction samples based on a sum of the prediction refinement and the samples of the two corresponding regions and a clipping operation; and

performing the conversion based on the final prediction samples,

wherein the clipping operation is within a range [minPred, maxPred], and wherein the maxPred is based on a sample bit-depth of the current block, the minPred and the maxPred are integers.

6 . The method of claim 1 , wherein T 0 is signaled in a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a tile group header, a tile, a coding unit (CU) or a coding tree unit (CTU).

7 . The method of claim 1 , wherein a bi-linear filter is used in the first interpolation filtering operation, or

wherein only an 8-tap interpolation filter is used in the second interpolation filtering operation for the luma component of the current block.

8 . The method of claim 1 , wherein a fourth number of samples that comprise a difference between the second number of samples and the third number of samples are padded in the padding operation, or

wherein L1=8 and G1=2, or

wherein k1=0, or

wherein M1 is equal to 8 or 16, and N1 is equal to 8 or 16.

9 . The method of claim 1 , wherein the conversion includes encoding the current block into the bitstream.

10 . The method of claim 1 , wherein the conversion includes decoding the current block from the bitstream.

11 . An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:

determine, for a conversion between a current block of a video and a bitstream of the video, two corresponding regions for a sub-block of the current block, wherein the two corresponding regions are in two reference pictures of the current block respectively;

derive a sum of absolute difference (SAD) based on partial samples of the two corresponding regions, wherein positions of the partial samples of the two corresponding regions are predetermined,

determine that a first prediction mode is applied to the sub-block when the SAD is larger than or equal to a first threshold, wherein in the first prediction mode, prediction samples of the current block is refined at least based on spatial gradient values in different directions, wherein the first prediction mode is a bi-directional optical flow (BDOF) prediction mode; and

derive the spatial gradient values in different directions based on samples of the two corresponding regions and an arithmetic shifting operation with a shift value S which is a fixed integer and is not equal to 4, wherein the spatial gradient values are derived based on samples from a same reference picture,

wherein whether the first prediction mode being applied to the sub-block is further determined based on one or more picture order count (POC) parameters associated with a current picture comprising the current block of the video,

wherein a first reference picture (R 0 ) and a second reference picture (R 1 ) of the two reference pictures are associated with the current picture, wherein τ 0 is a difference between a picture order count of the current picture and a picture order count of the first reference picture (τ 0 =POC(current)−POC(R 0 )), and wherein τ 1 is a difference between a picture order count of the second reference picture and the picture order count of the current picture (τ 1 =POC(R 1 )−POC(current)),

wherein the first prediction mode is disabled in response to abs (abs(τ 0 )−abs(τ 1 ))≥τ 0 , wherein T 0 is an integer, and T 0 is larger than 0; or wherein the first prediction mode is disabled in response to abs(τ 0 )≥abs(τ 1 )×T 0 or abs(τ 1 )≥abs(τ 0 )×T 0 , wherein T 0 is an integer, wherein T 0 =1, and

wherein in the first prediction mode, before deriving the spatial gradient values, one or more outer samples of the current block are derived with a padding operation instead of a first interpolation filtering operation, and a second interpolation filtering operation is performed for the current block;

wherein a size of the current block is M1×N1, wherein a first number of samples required by a gradient calculation is (M1+G1)× (N1+G1), wherein an interpolation filter used in the second interpolation filtering operation for a luma component of the current block comprises L1 taps, wherein a second number of samples required by the second interpolation filtering operation without enabling the first prediction mode is (M1+G1+L1−1)× (N1+G1+L1−1), wherein a third number of samples required by the second interpolation filtering operation with enabling the first prediction mode is (M1+k1+L1−1)× (N1+k1+L1−1), wherein M1, N1, G1, and L1 are positive integers, and wherein k1 is an integer less than G1.

12 . A method for storing a bitstream of a video, comprising:

determining, for a current block of the video, two corresponding regions for a sub-block of the current block, wherein the two corresponding regions are in two reference pictures of the current block respectively;

deriving a sum of absolute difference (SAD) based on partial samples of the two corresponding regions, wherein positions of the partial samples of the two corresponding regions are predetermined;

determining that a first prediction mode is applied to the sub-block when the SAD is larger than or equal to a first threshold, wherein in the first prediction mode, prediction samples of the current block is refined at least based on spatial gradient values in different directions, wherein the first prediction mode is a bi-directional optical flow (BDOF) prediction mode;

deriving the spatial gradient values in different directions based on samples of the two corresponding regions and an arithmetic shifting operation with a shift value S which is a fixed integer and is not equal to 4, wherein the spatial gradient values are derived based on samples from a same reference picture;

generating the bitstream; and

storing the bitstream in a non-transitory computer-readable recording medium,

wherein whether the first prediction mode being applied to the sub-block is further determined based on one or more picture order count (POC) parameters associated with a current picture comprising the current block of the video,

wherein a first reference picture (R 0 ) and a second reference picture (R 1 ) of the two reference pictures are associated with the current picture, wherein τ 0 is a difference between a picture order count of the current picture and a picture order count of the first reference picture (τ 0 =POC(current)−POC(R 0 )), and wherein τ 1 is a difference between a picture order count of the second reference picture and the picture order count of the current picture (τ 1 =POC(R 1 )−POC(current)),

wherein the first prediction mode is disabled in response to abs (abs(τ 0 )−abs(τ 1 ))≥τ 0 , wherein T 0 is an integer, and T 0 is larger than 0; or wherein the first prediction mode is disabled in response to abs(τ 0 )≥abs(τ 1 )×T 0 or abs(τ 1 )≥abs(τ 0 )×τ 0 , wherein T 0 is an integer, wherein T 0 =1, and

wherein in the first prediction mode, before deriving the spatial gradient values, one or more outer samples of the current block are derived with a padding operation instead of a first interpolation filtering operation, and a second interpolation filtering operation is performed for the current block;

wherein a size of the current block is M1×N1, wherein a first number of samples required by a gradient calculation is (M1+G1)× (N1+G1), wherein an interpolation filter used in the second interpolation filtering operation for a luma component of the current block comprises L1 taps, wherein a second number of samples required by the second interpolation filtering operation without enabling the first prediction mode is (M1+G1+L1−1)× (N1+G1+L1−1), wherein a third number of samples required by the second interpolation filtering operation with enabling the first prediction mode is (M1+k1+L1−1)× (N1+k1+L1−1), wherein M1, N1, G1, and L1 are positive integers, and wherein k1 is an integer less than G1.

13 . A non-transitory computer-readable storage medium storing instructions that cause a processor to:

determine, for a conversion between a current block of a video and a bitstream of the video, two corresponding regions for a sub-block of the current block, wherein the two corresponding regions are in two reference pictures of the current block respectively;

derive a sum of absolute difference (SAD) based on partial samples of the two corresponding regions, wherein positions of the partial samples of the two corresponding regions are predetermined;

determine that a first prediction mode is applied to the sub-block when the SAD is larger than or equal to a first threshold, wherein in the first prediction mode, prediction samples of the current block is refined at least based on spatial gradient values in different directions, wherein the first prediction mode is a bi-directional optical flow (BDOF) prediction mode; and

derive the spatial gradient values in different directions based on samples of the two corresponding regions and an arithmetic shifting operation with a shift value S which is a fixed integer and is not equal to 4, wherein the spatial gradient values are derived based on samples from a same reference picture,

wherein whether the first prediction mode being applied to the sub-block is further determined based on one or more picture order count (POC) parameters associated with a current picture comprising the current block of the video,

wherein a first reference picture (R 0 ) and a second reference picture (R 1 ) of the two reference pictures are associated with the current picture, wherein τ 0 is a difference between a picture order count of the current picture and a picture order count of the first reference picture (τ 0 =POC(current)−POC(R 0 )), and wherein τ 1 is a difference between a picture order count of the second reference picture and the picture order count of the current picture (τ 1 =POC(R 1 )−POC(current)),

wherein the first prediction mode is disabled in response to abs (abs(τ 0 )−abs(τ 1 ))≥T 0 , wherein T 0 is an integer, and T 0 is larger than 0; or wherein the first prediction mode is disabled in response to abs(τ 0 )>abs(τ 1 )×T 0 or abs(τ 1 )>abs (τ 0 )×T 0 , wherein T 0 is an integer, wherein T 0 =1, and

wherein in the first prediction mode, before deriving the spatial gradient values, one or more outer samples of the current block are derived with a padding operation instead of a first interpolation filtering operation, and a second interpolation filtering operation is performed for the current block;

wherein a size of the current block is M1×N1, wherein a first number of samples required by a gradient calculation is (M1+G1)× (N1+G1), wherein an interpolation filter used in the second interpolation filtering operation for a luma component of the current block comprises L1 taps, wherein a second number of samples required by the second interpolation filtering operation without enabling the first prediction mode is (M1+G1+L1−1)× (N1+G1+L1−1), wherein a third number of samples required by the second interpolation filtering operation with enabling the first prediction mode is (M1+k1+L1−1)× (N1+k1+L1−1), wherein M1, N1, G1, and L1 are positive integers, and wherein k1 is an integer less than G1.

Assignments (2)

Priority Claims (1)

Continuity (4)

References Cited (102)