Temporal attention-based neural networks for video compression
Systems, methods, and instrumentalities are disclosed for video encoding and/or video decoding using artificial neural networks (e.g., convolutional neural networks or recurrent neural networks), attention, and/or attention with spatial attributes. For example, an apparatus may be configured to perform one or more of the following: obtaining a context block, a current block, and a latent vector associated with the context block; performing at least one convolution on the context block, the reference block, and the latent vector; generating motion flow data associated with the current block based on the at least one convolution; or generating a bitstream the comprises an indication of the motion flow data. The motion flow data may be quantized. The generated bitstream may comprise an indication of the quantized motion flow data.
1 . A video decoding device, comprising:
a processor configured to:
obtain encoded motion flow data associated with a current block;
obtain a motion latent vector based on an attention associated with motion information from at least one past block and based on state information associated with the at least one past block, wherein the at least one past block is a previously reconstructed block;
obtain decoded motion flow data based on the encoded motion flow data and the motion latent vector;
generate a prediction block for the current block based on the decoded motion flow data; and
reconstruct the current block based on the prediction block.
2 . The video decoding device of claim 1 , wherein the decoded motion data is obtained using a deconvolution on the encoded motion flow data and the motion latent vector.
3 . The video decoding device of claim 2 , wherein the deconvolution is performed using a recurrent neural network.
4 . The video decoding device of claim 1 , wherein the processor is further configured to:
concatenate the encoded motion flow data and the motion latent vector, wherein the decoded motion flow data is obtained based on the concatenation.
5 . The video decoding device of claim 1 , wherein the state information indicates a latent representation.
6 . The video decoding device of claim 1 , wherein the at least one past block is a first past block, and wherein the attention associated with motion information from the at least one past block is obtained further based on respective state information from the first past block and a second past block, wherein the each of the state information is associated with a respective latent representation.
7 . The video decoding device of claim 1 , wherein the processor is further configured to:
obtain a weighted attention value, wherein the motion latent vector is obtained further based on the weighted attention value.
8 . A video encoding device, comprising:
a processor configured to:
obtain a motion latent vector based on an attention associated with motion information from at least one past block, and based on state information associated with at least one previously encoded block;
obtain a reference block and a current block;
generate motion flow data associated with the current block based on the current block, the reference block, and the motion latent vector; and
generate video data that comprises an indication of the motion flow data.
9 . The video encoding device of claim 8 , wherein the motion flow data associated with the current block is generated further based on performing a convolution on the current block, the reference block, and the motion latent vector.
10 . The video encoding device of claim 9 , wherein the processor is further configured to:
generate, based on the convolution performed on the current block, a latent representation for the current block for generating motion flow data associated with a second block.
11 . The video encoding device of claim 8 , wherein the motion latent vector is configured to indicate an attention associated with the current block.
12 . The video encoding device of claim 8 , wherein the processor is further configured to:
perform spatial feature extraction, wherein the motion latent vector is obtained further based on the performed spatial feature extraction.
13 . A video decoding method, comprising:
obtaining encoded motion flow data associated with a current block;
obtaining a motion latent vector based on an attention associated with motion information from at least one past block and based on state information associated with the at least one past block, wherein the at least one past block is a previously reconstructed block;
obtaining decoded motion flow data based on the encoded motion flow data and the motion latent vector;
generating a prediction block for the current block based on the decoded motion flow data; and
reconstructing the current block based on the prediction block.
14 . The video decoding method of claim 13 , wherein the decoded motion data is obtained using a deconvolution on the encoded motion flow data and the motion latent vector, wherein the deconvolution is performed using a recurrent neural network.
15 . The video decoding method of claim 13 , wherein the method further comprises:
concatenating the encoded motion flow data and the motion latent vector, wherein the decoded motion flow data is obtained based on the concatenated encoded motion flow data and the motion latent vector.
16 . The video decoding method of claim 13 , wherein the at least one past block is a first past block, and wherein the attention associated with motion information from the at least one past block is obtained further based on respective state information from the first past block and a second past block, wherein each of the state information is associated with a respective latent representation.
17 . The video decoding method of claim 13 , wherein the method further comprises:
obtaining a weighted attention value, wherein the motion latent vector is obtained further based on the weighted attention value.