Inter-prediction method and apparatus using reference frame generated based on deep learning
An inter-prediction method and apparatus uses a reference frame generated based on deep learning. In the inter-prediction method and apparatus, a reference frame is selected, and a virtual reference frame is generated based on the selected reference frame. A reference picture list is configured to include the generated virtual reference frame, and inter prediction for a target block is performed based on the virtual reference frame. The virtual reference frame may be generated based on a deep-learning network architecture, and may be generated based on video interpolation and/or video extrapolation that use the selected reference frame.
1. A decoding method, comprising:
selecting a plurality of reference frames;
generating a virtual reference frame using a neural network to which the plurality of the reference frames are input; and
performing inter prediction based on the virtual reference frame, wherein
a time point of the virtual reference frame is different from time points of the plurality of the reference frames,
the neural network generates a residual frame which is a residual between the plurality of the reference frames of which time points are different,
the neural network generates the virtual reference frame based on a predicted feature vector of the residual frame, and
the predicted feature vector is generated by a convolutional Long Short-Term Memory (LSTM) network which performs a prediction for the predicted feature vector in a time series using feature vectors of the plurality of reference frames.
2. The decoding method of claim 1 ,
wherein the virtual reference frame is generated based on a sum of the predicted vector of the residual frame and a feature vector of a previously reconstructed frame.
3. The decoding method of claim 1 ,
wherein the neural network generates the feature vectors for the plurality of the reference frames, and generates the virtual reference frame based on the feature vectors, and
wherein each of the feature vectors is a vector representing features of a reference frame at a dimension lower than a dimension of the reference frame.
4. The decoding method of claim 1 ,
wherein the virtual reference frame is generated based on a Generative Adversarial Network (GAN) architecture.
5. The decoding method of claim 1 ,
wherein the virtual reference frame is generated based on an Adaptive Convolution Network (ACN) architecture.
6. The decoding method of claim 1 ,
wherein the virtual reference frame is generated via interpolation that uses frames predicted by network architectures.
7. The decoding method of claim 1 ,
wherein the virtual reference frame is generated based on video interpolation that uses frames predicted by the neural network using the plurality of the reference frames.
8. The decoding method of claim 7 ,
wherein the predicted frames includes a first predicted frame and a second predicted frame,
wherein the first predicted frame is predicted by the neural network using at least one reference frame of which time point is smaller than the time point of the virtual reference frame, and
wherein the second predicted frame is predicted by the neural network using at least one reference frame of which time point is greater than the time point of the virtual reference frame.
9. The decoding method of claim 1 ,
wherein the virtual reference frame is generated based on video extrapolation that uses frames predicted by the neural network using the plurality of the reference frames.
10. The decoding method of claim 1 , further comprising configuring a reference picture list based on the virtual reference frame.
11. The decoding method of claim 10 , wherein a specific reference frame, among reference frames included in a Decoded Picture Buffer (DPB), is replaced with the virtual reference frame.
12. The decoding method of claim 1 , wherein an inter-prediction mode for the inter prediction is an Advanced Motion Vector Prediction (AMVP) mode.
13. The decoding method of claim 1 ,
wherein an inter-prediction mode for the inter prediction is a merge mode or a skip mode.
14. The decoding method of claim 1 ,
wherein at least one of the plurality of the reference frames is a reference frame having a shortest distance from a target frame in a backward direction or a forward direction, among reference frames included in a reference picture list.
15. The decoding method of claim 1 ,
wherein, when one reference frame of the plurality of the reference frames is selected from each of two directions,
a first difference and a second difference are equal to each other,
the first difference is a difference between a Picture Order Count (POC) of the selected reference frame in a backward direction of the two directions and a POC of a target frame, and
the second difference is a difference between the POC of the target frame and a POC of the selected reference frame in a forward direction of the two directions.
16. The decoding method of claim 1 ,
wherein at least one of the plurality of the reference frames is a reference frame compressed using a smallest Quantization Parameter (QP), among reference frames included in a Decoded Picture Buffer (DPB).
17. The decoding method of claim 1 ,
wherein reference-frame specific information indicates the plurality of the reference frames that are used to generate the virtual reference frame, and are signaled for a specific unit.
18. The decoding method of claim 1 ,
wherein the plurality of the reference frames are selected based on temporal identifiers of the plurality of the reference frames.
19. An encoding method, comprising:
selecting a plurality of reference frames;
generating a virtual reference frame using a neural network to which the plurality of the reference frames are input; and
performing inter prediction based on the virtual reference frame, wherein
a time point of the virtual reference frame is different from time points of the plurality of the reference frames,
the neural network generates a residual frame which is a residual between the plurality of the reference frames of which time points are different,
the neural network generates the virtual reference frame based on a predicted feature vector of the residual frame, and
the predicted feature vector is generated by a convolutional Long Short-Term Memory (LSTM) network which performs a prediction for the predicted feature vector in a time series using feature vectors of the plurality of reference frames.
20. An inter-prediction method, comprising:
selecting a plurality of reference frames;
generating a virtual reference frame using a neural network to which the plurality of the reference frames are input; and
performing inter prediction based on the virtual reference frame, wherein
a time point of the virtual reference frame is different from time points of the plurality of the reference frames,
the neural network generates a residual frame which is a residual between the plurality of the reference frames of which time points are different,
the neural network generates the virtual reference frame based on a predicted feature vector of the residual frame, and
the predicted feature vector is generated by a convolutional Long Short-Term Memory (LSTM) network which performs a prediction for the predicted feature vector in a time series using feature vectors of the plurality of reference frames.