SYSTEMS, METHODS, AND BITSTREAM STRUCTURE FOR VIDEO CODING AND DECODING FOR MACHINES WITH ADAPTIVE INFERENCE
Systems and methods are provided for encoding and decoding video data for machine applications, such as Video Coding for Machines, using inference models. An encoder uses an inference selector to determine an appropriate inference model to encode a feature substream. The encoder further employs an inference metadata encoder to encode parameters of the selected inference model into an inference metadata substream, which can be multiplexed with the feature substream to generate an encoded bitstream to send to a decoder site. A decoder receiving the encoded bitstream, extracts the inference metadata, selects the appropriate inference model, and applies that inference model in decoding the feature substream and generating a decoded output signal for machine consumption.
1 . An encoder for video coding for machine applications, the encoder comprising:
an inference selector;
an inference metadata encoder coupled to the inference selector and receiving inference model selection parameters from the inference encoder and encoding said parameters into an inference metadata substream;
an inference encoder, the inference encoder receiving the input signal and inference model selection parameters from the inference selector and routing the input signal to a selected inference model;
a feature encoder, the feature encoder being coupled to the inference encoder and generating an encoded feature substream;
a multiplexor, the multiplexor receiving the inference metadata substream from the inference metadata encoder and the feature substream from the feature encoder and providing an encoded bitstream therefrom.
2 . The encoder of claim 1 , wherein the inference selector produces a recommendation for a best matching inference model for the input signal.
3 . The encoder of claim 2 , wherein the inference selector recommends an inference model for each unit of the input signal.
4 . The encoder of claim 3 , wherein the encoder comprises a plurality of inference models and the inference encoder operates to route each unit of the input signal to the recommended inference model for that unit.
5 . A decoder for video for video coding for machine applications encoded with an inference encoder, the decoder comprising:
a demultiplexor, the demultiplexor for receiving an encoded bitstream having features and inference metadata encoded therein, the demultiplexor generating a feature substream and an inference metadata substream;
an inference metadata decoder, the inference metadata decoder being coupled to the demultiplexor and receiving the inference metadata substream and extracting parameters of an inference model used to encode the bitstream therefrom;
an inference selector responsive to the inference model parameters and selecting an inference model from a plurality of inference models;
a feature decoder coupled to the demultiplexor, the feature decoder receiving the feature substream and extracting encoder features therefrom; and
an inference decoder, the inference decoder receiving the features from the feature decoder and the selected inference model from the inference selector and providing a decoded output signal therefrom.
6 . The decoder of claim 5 , wherein the bitstream comprises a stream level header, the stream level header having data used by the demultiplexor to extract the feature substream and inference metadata substream from the bitstream.
7 . The decoder of claim 5 , wherein the inference metadata substream further comprises an inference metadata header and an inference metadata payload,
8 . The decoder of claim 6 , wherein the inference metadata header 830 is used by the inference metadata decoder 686 to extract and decode the inference metadata payload.
9 . The decoder of claim 5 , wherein the feature substream comprises a feature stream header and a feature stream payload, wherein the feature stream header is used by the feature decoder to decode the feature stream payload.
10 . The decoder of claim 5 , wherein the inference selector produces a recommendation for a best matching inference model for the input signal.
11 . The decoder of claim 10 , wherein the inference selector recommends an inference model for each unit of the input signal.
12 . The decoder of claim 11 , wherein the encoder comprises a plurality of inference models and the inference encoder operates to route each unit of the input signal to the recommended inference model for that unit.
13 . A bitstream for image information encoded using an inference model comprising:
a stream level header;
a feature substream comprising a feature stream header and a feature stream payload; and
an inference metadata substream comprising an inference metadata header and an inference metadata payload.