Systems and methods for coding and decoding image data using general adversarial models
A video encoder and decoder system employing a compression-decision generative adversarial model (“CDGAM”) has an encoder with a compression network and a decoder with a task network. The compression network is a generator portion of a pretrained CDGAM and the task network is a discriminator portion of the pretrained CDGAM. The compression network and task network are trained using common image data wherein the compression network is trained to a generate minimal compressed representation that results in a substantially similar loss as the original image at the discriminator.
1 . A video encoder for a machine-video application comprising:
a preprocessor receiving an input signal, the preprocessor including at least a portion of a convolutional neural network outputting an intermediate feature map representing the input signal for a particular machine task;
a compression network coupled to the preprocessor, the compression network comprising a pretrained generator in a compression-decision generative adversarial model, the generator being trained to compress the intermediate feature map to a minimal compressed representation that results in an acceptably similar loss to a discriminator for the
machine task as the intermediate feature map; and
a video encoder coupled to the compression network and generating an encoded machine task bitstream.
2 . The video encoder of claim 1 , wherein the compression network is trained with a discriminator providing a generator loss function that is backpropagated to the generator.
3 . The video encoder of claim 2 , wherein, compression network is trained with a loss function:
L
C
D
=
E
x
[
log
D
(
x
)
]
+
E
x
[
log
(
1
-
D
(
C
(
x
)
)
)
]
subject to R(C(x))<R T ,
where C(x) is the output of the compression network, R(C(x)) is the rate in bits of the compressed representation, R T is a rate threshold, and D(x) is the output of the discriminator.
4 . The video encoder of claim 1 , wherein the unified bitstream comprises:
a stream level header;
a video substream having a video substream header and a video substream payload; and
a network update substream having a network update substream header and a network update payload.
5 . A video decoder for a machine-video application comprising:
a video decoder receiving an encoded machine-task bitstream representing intermediate feature layer for a machine task and outputting compressed intermediate feature data; and
a task network coupled to the video decoder and receiving the compressed intermediate feature data, the task network comprising a pretrained discriminator from a compression-decision generative adversarial model, wherein the task network is trained with intermediate feature data and a generator providing compressed intermediate feature data, the discriminator providing an output loss function of the intermediate feature data and the compressed intermediate feature data that is backpropagated to the generator during training to identify minimally compressed intermediate feature data with similar loss to the discriminator for the machine task as the input signal.
6 . The video decoder of claim 5 , wherein the task network is trained with a loss function:
L
C
D
=
E
x
[
log
D
(
x
)
]
+
E
x
[
log
(
1
-
D
(
C
(
x
)
)
)
]
subject to R(C(x))<R T ,
where C(x) is the output of the compression network, R(C(x)) is the rate in bits of the compressed representation, R T is a rate threshold, and D(x) is the output of the discriminator.