Systems and methods for video coding for machines using an autoencoder
Systems and methods for encoding and decoding video for machine consumption (video coding for machines) are provided in which an autoencoder is employed. The autoencoder has an encoder portion, a bottleneck portion, and a decoder portion. The autoencoder being distributed between a VCM encoder and VCM decoder such that the VCM encoder includes the encoder portion and bottleneck portion and the VCM decoder includes the bottleneck portion and the decoder portion.
1 . An encoder for video for video coding for machine applications, the encoder comprising:
an autoencoder encoder portion;
a latent feature encoder coupled to the autoencoder encoder portion; and
an autoencoder description encoder, the autoencoder description encoder being coupled to the autoencoder encoder portion;
a multiplexor, the multiplexor coupled to the latent feature encoder and the autoencoder description encoder and providing a coded bitstream.
2 . The encoder of claim 1 , wherein the autoencoder encoder portion is a neural network comprising a plurality of successive encoder layers and a bottle neck layer after the final encoder layer.
3 . The encoder of claim 2 , wherein with each successive encoder layer having fewer neurons than the prior layer, and a bottle neck layer having fewer neurons than the final encoder layer.
4 . The encoder of claim 3 , wherein the bottle neck layer has fewer neurons than the final encoder layer.
5 . The encoder of claim 2 , wherein the number of encoder layers is the same as a number of layers in a decoder portion of a compatible decoder.
6 . An decoder for encoded video for video coding for machine applications, the decoder comprising:
a demultiplexor receiving a bitstream encoded with an autoencoder;
a latent feature decoder coupled tot the demultiplexor;
an autoencoder description decoder coupled to the demultiplexor;
an autoencoder decoder portion, the autoencoder decoder portion coupled to the autoencoder description decoder and the latent feature decoder, the autoencoder decoder portion providing decoded video for machine use.
7 . The decoder of claim 5 , wherein the autoencoder decoder portion is a neural network comprising a bottle neck layer as an input layer and a plurality of successive decoder layers.
8 . The decoder of claim 6 , wherein with each successive decoder layer has a higher number of neurons than the prior layer.
9 . The decoder of claim 7 , wherein the bottle neck layer has fewer neurons than the first decoder layer.
10 . The decoder of claim 6 , wherein the number of decoder layers is the same as a number of layers in an encoder portion of the encoder which generated the coded bitstream.
11 . A system for encoding image data in a video coding for machine application, the system comprising:
a video coding for machine (VCM) encoder, the VCM encoder comprising an autoencoder encoder portion and an autoencoder bottleneck portion, the VCM encoder receiving image information and generating a coded bitstream including feature data of the image information; and
a VCM decoder, the VCM decoder comprising the autoencoder bottleneck portion and an autoencoder decoder portion, the VCM decoder receiving the coded bitstream and reconstructing the image information from the feature data.
12 . The system of claim 11 , wherein the autoencoder encoder portion is a neural network comprising a plurality of successive encoder layers with each successive encoder layer having fewer neurons than the prior layer, and the bottle neck portion having fewer neurons than a final encoder layer.
13 . The system of claim 12 , wherein the autoencoder decoder portion is a neural network comprising a plurality of successive encoder layers with each successive decoder layer having a higher number of neurons than the prior layer, and the bottle neck portion having fewer neurons than the first decoder layer.
14 . The system of claim 13 , wherein the number of encoder layers in the encoder portion is the same as a number of layers in a decoder portion.
15 . The system of claim 11 , wherein the autoencoder encoder portion is in communication with the autoencoder decoder portion and updates to the autoencoder are communicated between the VCM encoder and VCM decoder.
16 . The system of claim 11 , wherein the VCM encoder further comprises:
an autoencoder description encoder, the autoencoder description encoder being coupled to the autoencoder encoder portion;
a latent feature encoder coupled to the autoencoder encoder portion; and
a multiplexor, the multiplexor coupled to the latent feature encoder and the autoencoder description encoder and providing a coded bitstream.
17 . The system of claim 11 , wherein VCM decoder further comprises:
a demultiplexor receiving the coded bitstream;
a latent feature decoder coupled to the demultiplexor;
an autoencoder description decoder coupled to the demultiplexor; and
the autoencoder decoder portion being coupled to the autoencoder description decoder and the latent feature decoder.