Method for neural network-based video encoding and decoding, and video encoding apparatus
There is provided a method for neural network-based video encoding. The method includes estimating a motion vector between an input image and a reference image based on a temporal layer of the input image, transforming the motion vector into a latent representation, scaling the latent representation of the motion vector based on the temporal layer of the input image and obtaining a temporal context of the input image based on the scaled latent representation of the motion vector and the reference image.
1 . A method for neural network-based video encoding, the method comprising:
estimating a motion vector between an input image and a reference image based on a temporal layer of the input image;
transforming the motion vector into a latent representation;
scaling the latent representation of the motion vector based on the temporal layer of the input image; and
obtaining a temporal context of the input image based on the scaled latent representation of the motion vector and the reference image.
2 . The method of claim 1 , wherein the reference image comprises a first bidirectional reference image temporally before the input image and a second bidirectional reference image temporally after the input image.
3 . The method of claim 1 , wherein the scaling based on the temporal layer comprises scaling the latent representation of the motion vector by dividing the latent representation of the motion vector into quantization step determining parameters defined for the temporal layer.
4 . The method of claim 1 , further comprising:
performing entropy encoding and entropy decoding on the latent representation of the motion vector;
rescaling the scaled latent representation of the motion vector based on the temporal layer; and
reconstructing motion vectors based on the rescaled latent representation of the motion vector,
wherein the obtaining of the temporal context comprises obtaining the temporal context based on the reconstructed motion vectors and the reference image.
5 . The method of claim 4 , wherein the rescaling based on the temporal layer comprises multiplying the scaled latent representation of the motion vector by the quantization step determining parameters defined for the temporal layer.
6 . The method of claim 4 , wherein the obtaining of the temporal context comprises:
outputting a reference feature map by inputting the reference image into a feature extraction neural network,
performing bilinear warping on the reference feature map based on the reconstructed motion vectors to output a warped reference feature map,
inputting the warped reference feature map to a post-processing neural network, and
inputting an output of the post-processing neural network to a context fusion network to output the temporal context.
7 . A method for neural network-based video encoding, the method comprising:
estimating a motion vector between an input image and a reference image based on a temporal layer of the input image;
transforming the motion vector into a latent representation;
obtaining a temporal context of the input image based on the latent representation of the motion vector and the reference image; and
performing a smoothing operation on the temporal context based on a smoothing object comprising at least one of the reference image, the input image, the motion vector, an input in the obtaining of the temporal context, an output in the obtaining of the temporal context, or an input or output of a sub-process in the obtaining of the temporal context.
8 . The method of claim 7 , wherein the reference image comprises a first bidirectional reference image temporally before the input image and a second bidirectional reference image temporally after the input image.
9 . An electronic device comprising:
a memory configured to store one or more instructions and a reference image; and
a processor configured to execute the one or more instructions to: estimate a motion vector between an input image and the reference image based on the reference image, the input image, and a temporal layer of the input image;
transform the motion vector into a latent representation;
scale the latent representation of the motion vector based on the temporal layer of the input image; and
obtain a temporal context of the input image based on the scaled latent representation of the motion vector and the reference image.