Video encoder, video encoding method, and video decoder
A video encoder is provided. The video encoder according to an example embodiment includes: a differentiable prediction (DP) module configured to output an optimal initial search position by performing full search in a predetermined area by using a pair of frames of a video as input; and a motion estimation (ME) module configured to perform motion estimation by moving a search position toward the optimal initial search position output by the DP module.
1 . A video encoder comprising:
a differentiable prediction (DP) module configured to output an initial search position in a video by performing full search in a predetermined area of the video by using a pair of frames of the video as input to the DP module; and
a motion estimation (ME) module configured to perform motion estimation by moving a search position toward the initial search position output by the DP module.
2 . The video encoder of claim 1 , wherein the DP module is configured to generate a predicted image for each of a plurality of initial search positions in the predetermined area, and configured to output an initial search position, at which a residual between the generated predicted image and a first frame of the pair of frames is minimized, as the initial search position.
3 . The video encoder of claim 2 , wherein the DP module comprises:
an affine transformation module configured to perform affine transformation on a second frame of the pair of frames for each of the plurality of initial search positions; and
a motion estimation motion compensation (MEMC) module configured to:
perform motion estimation on the second frame, which is affine transformed for each of the plurality of initial search positions, to output motion in a kernel form; and
perform motion compensation based on the motion in the kernel form, to generate the predicted image.
4 . The video encoder of claim 3 , wherein the MEMC module is configured to:
for each block of a plurality of blocks of the affine-transformed second frame, perform unfolding of a block to divide the block into a plurality of patches, and calculate a sum of absolute differences (SAD) between the plurality of patches of the block and a corresponding block of the first frame; and
generate the motion in the kernel form based on calculated SADs of the plurality of blocks, by using softmax.
5 . The video encoder of claim 1 , wherein the DP module is configured to perform parallel processing on a plurality of pairs of frames of the video by using one or more processors.
6 . The video encoder of claim 5 , wherein the one or more processors comprise a graphic processing unit (GPU).
7 . The video encoder of claim 1 , further comprising a scaler configured to scale the pair of frames of the video to a size to be processed by the DP module.
8 . The video encoder of claim 1 , wherein at least one of a number and a size of the predetermined area is preset based on at least one of computing power, target processing speed, or accuracy of the motion estimation.
9 . A video encoding method comprising:
outputting, by a differentiable prediction (DP) module, an initial search position in a video by performing full search in a predetermined area of the video by using a pair of frames of the video as input to the DP module; and
performing, by a motion estimation (ME) module, motion estimation by moving a search position toward the initial search position output by the DP module.
10 . The video encoding method of claim 9 , wherein the outputting of the initial search position comprises:
generating a predicted image for a plurality of initial search positions in the predetermined area; and
outputting an initial search position, at which a residual between the generated predicted image and a first frame of the pair of frames is minimized, as the initial search position.
11 . The video encoding method of claim 10 , wherein the generating of the predicted image comprises:
performing affine transformation on a second frame of the pair of frames for each of the plurality of initial search positions;
performing motion estimation on the second frame, which is affine transformed for each of the plurality of initial search positions, to output motion in a kernel form; and
performing motion compensation based on the motion in the kernel form to generate the predicted image.
12 . The video encoding method of claim 11 , wherein the outputting of the motion in the kernel form comprises:
for each block of a plurality of blocks of the affine-transformed second frame, performing unfolding of a block to divide the block into a plurality of patches, and calculating a sum of absolute differences (SAD) between the plurality of patches of the block and a corresponding block of the first frame; and
generating the motion in the kernel form based on calculated SADs of the plurality of blocks, by using softmax.
13 . The video encoding method of claim 9 , wherein the outputting of the initial search position comprises performing parallel processing on a plurality of pairs of frames of the video by using one or more processors.
14 . The video encoding method of claim 9 , further comprising scaling the pair of frames of the video to a size to be processed by the DP module.
15 . The video encoder of claim 1 , wherein the pair of frames comprises a first frame and a second frame corresponding to a time preceding the first frame, and
wherein the DP module is further configured to:
perform an affine transformation on the second frame based on initial search positions to generate an affine-transformed second frame;
divide the first frame and the affine-transformed second frame into a plurality of first blocks and a plurality of second blocks, respectively, wherein each of the plurality of second blocks has a size that overlaps with neighboring second blocks; and
perform the motion estimation between the plurality of first blocks and the plurality of second blocks to output motion information.