Inter prediction method based on variable coefficient deep learning
An inter prediction method allows a variable coefficient deep learning model to adaptively learn characteristics of a video; transmits a variable coefficient deep learning model parameter generated from the learning from an image encoding device to an image decoding device; and refers to a virtual reference frame generated by the variable coefficient deep learning model.
1. A video decoding method performed by a video decoding apparatus, the video decoding method comprising:
decoding variable coefficient values, an affine prediction flag, and an encoding mode from a bitstream, the affine prediction flag indicating whether affine motion prediction is applied to a current block, and the encoding mode being an encoding mode for motion information of the current block and indicating a merge mode or an advanced motion vector prediction (AMVP) mode;
generating a virtual reference frame based on reference frames by using an interpolation model, wherein a variable coefficient network included in the interpolation model is set with the variable coefficient values; and
generating a merge candidate of the current block based on the virtual reference frame and the reference frames when the encoding mode is the merge mode.
2. The video decoding method of claim 1 , wherein the interpolation model is one of:
a deep learning model configured to directly generate the virtual reference frame from the reference frames;
a deep learning model configured to generate a mask from the reference frames and then perform weighed sum on the reference frames based on the mask to generate the virtual reference frame; or
a deep learning model configured to generate an optical flow from the reference frames and then warp the reference frames based on the optical flow to generate the virtual reference frame.
3. The video decoding method of claim 1 , wherein the variable coefficients values are transmitted from a video encoding apparatus and decoded by the video decoding apparatus, wherein the variable coefficients values are generated in advance by the video encoding apparatus updating the variable coefficient network of the interpolation model using a loss function based on a difference between an inferred frame generated by the interpolation model and an original frame.
4. The video decoding method of claim 1 , wherein the interpolation model further includes a fixed coefficient network, and
wherein fixed coefficient values of the fixed coefficient network are generated under pre-training based on an entire original reference frame and are set according to a predefined agreement.
5. The video decoding method of claim 4 , wherein the variable coefficient network of the interpolation model is connected to a back end of the fixed coefficient network.
6. The video decoding method of claim 1 , wherein the variable coefficient values are decoded per frame or at least one group of picture (GOP).
7. The video decoding method of claim 1 , wherein the generating of the merge candidates includes:
searching for spatial merge candidates; and
searching for a temporal merge candidate in consideration of the virtual reference frame.
8. The video decoding method of claim 7 , wherein the searching for the temporal merge candidate includes setting a zero motion vector as the merge candidate when both 1) a reference frame of the temporal merge candidate and 2) a picture in which there is a co-located block of the current block are the virtual reference frames.
9. The video decoding method of claim 1 , further comprising:
generating an affine AMVP candidate when the affine prediction flag is true and the encoding mode is the AMVP mode,
wherein the generating of the affine AMVP candidate includes:
adding an inherited affine AMVP candidate in consideration of the virtual reference frame; and
adding a constructed affine AMVP candidate in consideration of the virtual reference frame.
10. The video decoding method of claim 9 , wherein the adding of the inherited affine AMVP candidate includes adding a zero motion vector as the affine AMVP candidate when a reference picture of the inherited affine AMVP candidate and the reference picture of the current block are the same reference picture, and when the same reference picture is the virtual reference frame.
11. The video decoding method of claim 9 , wherein the adding of the constructed affine AMVP candidate includes adding a zero motion vector as a reference picture when a reference picture of all control point motion vectors of the constructed affine AMVP candidate and the reference picture of the current block are the same reference picture, and when the same reference picture is the virtual reference frame.
12. A video encoding method performed by a video encoding apparatus, the video encoding method comprising:
acquiring previously generated variable coefficient values and a preset encoding mode, the encoding mode being an encoding mode for motion information of a current block and indicating a merge mode or an advanced motion vector prediction (AMVP) mode;
generating a virtual reference frame from reference frames by using an interpolation model, wherein a variable coefficient network included in the interpolation model is set with the variable coefficient values; and
generating a merge candidate of the current block based on the virtual reference frame and the reference frames when the encoding mode is the merge mode.
13. The video encoding method of claim 12 , wherein the variable coefficient values are generated by updating the variable coefficient network of the interpolation model, and
wherein updating of the variable coefficient network uses a loss function based on a difference between an inferred frame generated by the interpolation model and an original frame.
14. The video encoding method of claim 12 , wherein the interpolation model further includes a fixed coefficient network, and fixed coefficient values of the fixed coefficient network are generated under pre-training based on an entire original reference frame and are set according to a predefined agreement.
15. The video encoding method of claim 12 , wherein the variable coefficient values are encoded per frame or at least one group of picture (GOP).
16. A method for providing video data to a video decoding device, the method comprising:
encoding the video data into a bitstream; and
transmitting the bitstream to the video decoding device,
wherein encoding the video data comprises:
acquiring previously generated variable coefficient values and a preset encoding mode from a bitstream, the encoding mode being an encoding mode for motion information of a current block and indicating a merge mode or an advanced motion vector prediction (AMVP) mode;
generating a virtual reference frame from reference frames using an interpolation model, wherein a variable coefficient network included in the interpolation model is set with the variable coefficient values; and
generating a merge candidate of the current block based on the virtual reference frame and the reference frames when the encoding mode is the merge mode.