Video and feature coding for multi-task machine learning
A system and method for video and feature coding of neural-network structures used for multi-task machine learning includes an encoder, decoder and a decoder-compliant bitstream. A task-specific video decoder includes a first decoder receiving a bitstream having at least one feature and a description of a neural network backbone used to generate the bitstream, and a task-specific neural network head. The neural network head recreates the neural network backbone from the description, receives a feature from the bitstream a generates a task-specific output.
1 . A machine video decoder for a machine video task comprising:
a first decoder receiving a bitstream, the bitstream having at least one feature map extracted from a video signal by a neural network backbone at an encoder and a description of the neural network backbone, the neural network backbone being a first part of a convolutional neural network and comprising a first set of convolution layers and a first set of pooling layers, the first decoder being one of an AVC decoder, an HEVC decoder and a VVC decoder and providing the at least one feature map and backbone description as outputs;
a neural network head comprising a second part of the convolutional neural network, the neural network head being trained for a specific machine task, the neural network head receiving the backbone description and at least one feature map from the first decoder and generating a task-specific output for the machine video task, wherein the neural network head comprises a deep neural network.
2 . The machine video decoder of claim 1 , further comprising a plurality of neural network heads, each of said neural network heads being trained for a specific task and receiving the at least one feature map and the output of the neural network backbone and generating a task-specific output.
3 . The machine video decoder of claim 1 wherein the second part of the convolutional neural network further comprises a second set of one or more convolution layers and a second set of one or more pooling layers.
4 . The machine video decoder of claim 1 wherein the bitstream includes a feature sequence parameter set containing first information about feature maps, a plurality of feature picture parameter sets containing second information about the feature maps, and a plurality of feature picture headers containing third information about the feature maps.
5 . The machine video decoder of claim 1 wherein the bitstream contains an SEI message containing information about the first part of the convolutional neural network.
6 . The machine video decoder of claim 1 wherein a split point between the first part of the convolutional neural network and the second part of the convolutional neural network is adaptively selected.
7 . The machine video decoder of claim 1 wherein the bitstream contains information about the size and position of the feature maps.
8 . A method for decoding an encoded bitstream for a machine video task comprising:
receiving a bitstream encoded using one of an AVC, HEVC, or VVC compliant encoding protocol, the bitstream including a sequence of feature maps extracted from a source video by an encoder using a first part of a convolutional neural network comprising a first set of one or more convolution layers and a first set of one or more pooling layers, and
decoding the bitstream with one of an AVC decoder, an HEVC decoder, or a VVC decoder and outputting the sequence of feature maps; and
applying the sequence of feature maps from the first part of the convolutional neural network to a second part of the convolutional neural network, the second part of the convolutional neural network completing the machine video task, wherein the second part of the convolutional neural network comprises a deep neural network.
9 . The decoding method of claim 8 wherein the second part of the convolutional neural network further comprises a second set of one or more convolution layers and a second set of one or more pooling layers.
10 . The decoding method of claim 8 wherein the bitstream includes a feature sequence parameter set containing first information about feature maps, a plurality of feature picture parameter sets containing second information about the feature maps, and a plurality of feature picture headers containing third information about the feature maps.
11 . The decoding method of claim 8 wherein the bitstream contains an SEI message containing information about the first part of the convolutional neural network.
12 . The decoding method of claim 8 wherein the bitstream includes information about the first part of the convolutional neural network.
13 . The decoding method of claim 9 wherein a split point between the first part of the convolutional neural network and the second part of the convolutional neural network is adaptively selected.
14 . The decoding method of claim 8 wherein the machine task is a machine vision task.
15 . The decoding method of claim 8 , wherein the machine vision task is one of detecting a class of an object, tracking an object, and object segmentation.
16 . The decoding method of claim 8 further comprising outputting the feature maps to a plurality of second parts of a convolutional neural network, each second part for completing a different machine video task.
17 . The decoding method of claim 8 wherein the deep neural network is a fully connected neural network.
18 . The decoding method of claim 8 wherein the bitstream contains information about the size and position of the feature maps.
19 . A machine video encoder for a machine video task, the encoder comprising:
a feature map extractor, the feature map extractor being a first part of a convolutional neural network and comprising a first set of convolution layers and a first set of pooling layers, the feature map extractor outputting a sequence of feature maps extracted from an input source video, and
an encoder encoding the extracted feature maps using one of an AVC, an HEVC encoder, or a VVC encoding protocol to generate an encoded bitstream for a machine video task to be completed at a decoding site having a second part of the convolutional neural network.
20 . The machine video encoder of claim 19 wherein the bitstream includes a feature sequence parameter set containing first information about the feature maps, a plurality of feature picture parameter sets containing second information about the feature maps, and a plurality of feature picture headers containing third information about the feature maps.
21 . The machine video encoder of claim 20 wherein the bitstream contains an SEI message containing information about the first part of the convolutional neural network.
22 . The machine video encoder of claim 20 wherein the bitstream includes information about the first part of the convolutional neural network.
23 . The encoder of claim 20 wherein a split point between the first part of the convolutional neural network and the second part of the convolutional neural network is adaptively selected.
24 . The encoder of claim 20 wherein the machine task is one of detecting a class of an on object, tracking an object, or object segmentation.