Temporal structure-based conditional convolutional neural networks for video compression
Video encoding and decoding is implemented with auto encoders using luminance information to derive motion information for chrominance prediction. In one embodiment conditional convolutions are used to encode motion flow information. A current condition, for example, GOP structure, is used as input to a succession of fully connected layers to implement the conditional convolution. In a related embodiment, more than one reference frame is used to encode motion flow information.
1 . A method, comprising:
receiving an input to a conditional convolution layer, wherein the input comprises a concatenated tensor of a current block and at least one reference block, and wherein the conditional convolution layer is at least one layer in a series of fully connected convolution layers;
processing at least one conditional convolution on the input, wherein the at least one conditional convolution is processed based at least on data representative of a current condition, and wherein the current condition comprises temporal-structure information associated with a group of pictures (GOP) structure;
encoding motion flow using an output from the at least one conditional convolution; and
generating a bitstream comprising the encoded motion flow.
2 . The method of claim 1 , wherein the data representative of the current condition that comprises the temporal-structure information associated with the GOP structure indicates at least one of: a framerate, a number of frames separating a first reference frame and a second reference frame, a distance between a current frame and a reference frame, a display order indicating whether the reference frame is before or after the current frame, a number of frames used to predict the current frame, a prediction direction between the current block and the reference block, or a content type.
3 . The method of claim 1 , wherein the GOP structure comprises a content type, and wherein the content type is at least one of: gaming, VR360 content, or screen content.
4 . The method of claim 1 , wherein the current condition is indicated by a one-hot encoded vector over a plurality of temporal conditions.
5 . The method of claim 1 , wherein the method further comprises:
indicating, in the bitstream, the data representative of the current condition that comprises the temporal-structure information associated with the GOP structure.
6 . The method of claim 1 , further comprising:
determining a scaling set associated with the conditional convolution layer, wherein the scaling set is determined from the data representative of the current condition that comprises the temporal-structure information associated with the GOP structure; and
processing the at least one conditional convolution on the input further by applying, via the conditional convolution layer, a weight and the scaling set to the input.
7 . The method of claim 6 , wherein the data representative of the current condition that comprises the temporal-structure information associated with the GOP structure indicates a framerate and a number of frames separating a first reference frame and a second reference frame, and wherein the scaling set is determined based on the framerate and the number of frames.
8 . The method of claim 1 , further comprising:
determining a bias vector associated with the conditional convolution layer, wherein the bias vector is determined from the data representative of the current condition that comprises the temporal-structure information associated with the GOP structure; and
processing the at least one conditional convolution on the input further by applying, via the conditional convolution layer, a weight and the bias vector to the input.
9 . The method of claim 8 , wherein the data representative of the current condition that comprises the temporal-structure information associated with the GOP structure indicates a framerate and a number of frames separating a first reference frame and a second reference frame, and wherein the bias vector is determined based on the framerate and the number of frames.
10 . An encoding device, comprising:
a processor, configured to:
receive an input to a conditional convolution layer, wherein the input comprises a concatenated tensor of a current block and at least one reference block, and wherein the conditional convolution layer is at least one layer in a series of fully connected convolution layers;
process at least one conditional convolution on the input, wherein the at least one conditional convolution is processed based at least on data representative of a current condition, wherein the current condition comprises temporal-structure information associated with a group of pictures (GOP) structure;
encode motion flow using an output from the at least one conditional convolution; and
generate a bitstream comprising the encoded motion flow.
11 . A method, comprising:
entropy decoding a bitstream comprising motion flow data associated with a current block;
processing at least one conditional deconvolution on the motion flow data via a conditional deconvolution layer to generate a reconstructed residue, wherein the conditional deconvolution layer is at least one layer in a series of fully connected layers, and wherein the at least one conditional deconvolution is processed based at least on data representative of a current condition, wherein the current condition comprises temporal-structure information associated with a group of pictures (GOP) structure; and
decoding the current block based on the reconstructed residue and a prediction of the current block.
12 . The method of claim 11 , wherein the data representative of the current condition that comprises the temporal-structure information associated with the GOP structure indicates at least one of: a framerate, a number of frames separating a first reference frame and a second reference frame, a distance between a current frame and a reference frame, a display order indicating whether the reference frame is before or after the current frame, a number of frames used to predict the current frame, a prediction direction between the current block and the reference block, or a content type.
13 . The method of claim 11 , wherein the GOP structure comprises a content type, and wherein the content type is at least one of: gaming, VR360 content, or screen content.
14 . The method of claim 11 , wherein the current condition is indicated by a one-hot encoded vector over a plurality of temporal conditions.
15 . The method of claim 11 , wherein the method further comprises:
receiving, in the bitstream, an indication of the data representative of the current condition that comprises the temporal-structure information associated with the GOP structure.
16 . The method of claim 11 , further comprising:
determining a scaling set associated with the conditional deconvolution layer, wherein the scaling set is determined from the data representative of the current condition that comprises the temporal-structure information associated with the GOP structure; and
processing the at least one conditional deconvolution on the motion flow data further by applying, via the conditional deconvolution layer, a weight and the scaling set to the motion flow data.
17 . The method of claim 16 , wherein the data representative of the current condition that comprises the temporal-structure information associated with the GOP structure indicates a framerate and a number of frames separating a first reference frame and a second reference frame, and wherein the scaling set is determined based on the framerate and the number of frames.
18 . The method of claim 11 , further comprising:
determining a bias vector associated with the conditional deconvolution layer, wherein the bias vector is determined from the data representative of the current condition that comprises the temporal-structure information associated with the GOP structure; and
processing the at least one conditional deconvolution on the motion flow data further by applying, via the conditional deconvolution layer, a weight and the bias vector to the motion flow data.
19 . The method of claim 18 , wherein the data representative of the current condition that comprises the temporal-structure information associated with the GOP structure indicates a framerate and a number of frames separating a first reference frame and a second reference frame, and wherein the bias vector is determined based on the framerate and the number of frames.
20 . A decoding device, comprising:
a processor, configured to:
entropy decode a bitstream comprising motion flow data associated with a current block;
process at least one conditional deconvolution on the motion flow data via a conditional deconvolution layer to generate a reconstructed residue, wherein the conditional deconvolution layer is at least one layer in a series of fully connected layers, and wherein the at least one conditional deconvolution is processed based at least on data representative of a current condition, wherein the current condition comprises temporal-structure information associated with a group of pictures (GOP) structure; and
decode the current block based on the reconstructed residue and a prediction of the current block.