Systems, methods and bitstream structure for hybrid feature video bitstream and decoder
Methods and systems for hybrid feature video bitstream encoding and decoding are provided. Encoding can be performed in one of a plurality of modes. In one mode, the video components of the encoded bitstream include residual data generated encoding the difference between feature data and input video data. A compatible decoder can operate in a plurality of decoding modes. In one decoding mode, the video components are decoded using the encoded video and encoded feature data in the bitstream. The hybrid bitstream between the encoder and decoder is formed with a plurality of hybrid segments, each having a plurality of components specifying characteristics the feature and video content of the segment.
1 . A hybrid decoder for a video bitstream supporting a video component and a feature component, comprising:
a demultiplexer, the demultiplexer receiving a hybrid bit stream and parsing the received bitstream into video components and feature components; the hybrid bitstream further comprising a plurality of segments, each segment comprising a plurality of components defining the feature and video components therein including a hybrid size component;
a video decoder coupled to the demultiplexer and receiving the video components of the bitstream;
a feature decoder coupled to the demultiplexer and receiving the feature components of the bitstream, the feature decoder being further coupled to the video decoder and selectively provides feature data thereto to facilitate video decoding;
a machine model coupled to the feature decoder;
the hybrid decoder having a first decoding mode determined by the decoder based on attributes of the bitstream for independently decoding the video components and feature components of the bitstream and a second decoding mode for decoding video components in the bitstream using, in part, feature components in the bitstream, wherein the decoding mode is determined at least in part by characteristics of the hybrid size component.
2 . The hybrid decoder of claim 1 , wherein the video components in the bitstream comprise residual data obtained by encoding the difference between feature data and input video data, and wherein the feature decoder provides feature data to the video decoder in the second decoding mode.
3 . The hybrid decoder of claim 1 , wherein each segment is separately decoded and the decoding mode is determined for each segment.
4 . A hybrid encoder for generating a hybrid bitstream comprising:
a preprocessor, the preprocessor receiving an input video stream and multiplexing the video stream for video processing and feature processing;
a video encoder, the video encoder compressing and encoding the video content for human viewing for inclusion in the bitstream,
a feature extractor, the feature extractor coupled to a machine model providing parameters for feature extraction for subsequent machine processing, the feature extractor being operatively coupled to the video encoder and selectively providing feature data thereto;
a feature encoder receiving feature data from the feature extractor and encoding said feature data for inclusion in a hybrid bitstream;
an optimizer coupled to the video encoder and the feature encoder, the optimizer further encoding the video and feature data for inclusion in the bitstream; and
a multiplexer, the multiplexer being coupled to the video encoder and feature encoder and generating a hybrid bitstream therefrom.
5 . The hybrid encoder of claim 4 , wherein the encoder operates in at least a first encoding mode for independently encoding the video components and feature components of the bitstream and a second encoding mode for encoding the video components in the bitstream using, in part, feature components from the feature encoder.
6 . The hybrid encoder of claim 5 , wherein in the second encoding mode, the video encoder generates residual data by encoding the difference between feature data and input video data.
7 . The hybrid encoder of claim 1 , wherein the hybrid bitstream comprises plurality of hybrid bitstream segments, each bitstream segment comprising a plurality of components including:
a hybrid size component;
a metadata component;
a feature header;
a feature payload;
a video header; and
a video payload.