Video codec importance indication and radio access network awareness configuration
Apparatuses, methods, and systems are disclosed for video codec importance indication and RAN awareness configuration. An apparatus includes a processor that detects a plurality of video coded network abstraction layer (“NAL”) units of a video coded stream, extracts semantic information associated with the plurality of the NAL units, combines the extracted semantic information associated with the plurality of NAL units to form a plurality of feature sets that is correspondingly synchronized with the plurality of NAL units that enclose the extracted semantic information, determines an information-to-importance value for each of the plurality of NAL units based on the plurality of feature sets without performing video decoding of the video coded stream, and indicates the determined information-to-importance value for each of the plurality of NAL units of the determined video codec specification to a video coded traffic-aware transceiver for scheduling video traffic based on the indicated information-to-importance value.
1. A network device apparatus, the apparatus comprising:
at least one memory; and
at least one processor coupled with the at least one memory and configured to cause the apparatus to:
detect a plurality of video coded network abstraction layer (“NAL”) units of a video coded stream according to defined syntax elements of a determined video codec specification;
extract semantic information associated with the plurality of NAL units according to a syntax elements and semantic knowledge base defined within the determined video codec specification;
combine the extracted semantic information associated with the plurality of NAL units to form a plurality of feature sets comprising a plurality of NAL unit type parameters, the plurality of feature sets correspondingly synchronized with the plurality of NAL units that enclose the extracted semantic information;
determine an information-to-importance value for each of the plurality of NAL units based on a NAL unit type, a NAL unit size, and a hierarchical encoding of each of the plurality of feature sets without performing video decoding of the video coded stream; and
indicate the determined information-to-importance value for each of the plurality of NAL units of the determined video codec specification to a video coded traffic-aware transceiver for scheduling video traffic based on the indicated information-to-importance value.
2. The apparatus of claim 1 , wherein the at least one processor is configured to cause the apparatus to parse the plurality of detected video coded NAL units as syntax elements of the determined video codec specification for extracting the semantic information.
3. The apparatus of claim 1 , wherein the at least one processor is configured to cause the apparatus to select a video codec syntax and semantic filter from a set of video codec syntax and semantic filter candidates based on the determined video codec specification and exposing the syntax elements and semantic knowledge base of the video codec specification.
4. The apparatus of claim 3 , wherein the at least one processor is configured to cause the apparatus to perform intrinsic semantic information extraction by applying the selected video codec syntax and semantic filter to at least one of a NAL unit's header and a NAL unit's non-entropy coded payload to extract the semantic information contained within the video coded stream.
5. The apparatus of claim 4 , wherein the intrinsic semantic information extraction is based on processing at least one NAL unit enclosed element selected from a group of:
video layer metadata information corresponding to one or more of a video layer identifier and a video layer priority information;
video layer frames-per-second playback metadata information;
video layer additional metadata for parsing layer-common video coded sequence information;
video sequence resolution corresponding to at least a combination of width and height of a video coded frame of the sequence of video coded frames;
video sequence chroma subsampling information;
video sequence coding metadata information corresponding to one or more of a coding block size, a number of coded rows, and a number of coded columns within the sequence of video coded frames;
video sequence number of maximally allowed reference frames for inter-frame motion prediction;
video sequence additional metadata necessary for parsing sequence-common video coded picture information;
video coded picture encoding mode configuration;
video coded picture segmentation information into picture partitions;
video coded picture additional metadata for parsing slice-common video coded slice information;
video coded slice type;
video coded slice payload size;
video coded slice position within one or more of a video coded picture, a tile, and a sub-picture;
video coded slice number of enclosed video coded rows;
video coded slice payload size of each enclosed video coded row; and
video coded slice reference list wherein the video coded reference list defines at least one reference slice for an inter-slice motion prediction and video encoding for a current slice enclosed within the NAL unit.
6. The apparatus of claim 5 , wherein the at least one processor is configured to cause the apparatus to extend the extracted intrinsic semantic information using an extrinsic source of semantic information synchronized with the plurality of NAL units of the video coded stream as provided by a video encoding function implementing the video codec specification that generated the plurality of NAL units.
7. The apparatus of claim 6 , wherein at least one of the extracted intrinsic semantic information and extrinsic semantic information of the determined video codec is at least one of combined, reduced, and post-processed into a plurality of normalized feature sets that are synchronized with the plurality of NAL units of the video coded stream.
8. The apparatus of claim 1 , wherein the determination of the information-to-importance value for each of the plurality of NAL units of the determined video codec specification comprises processing a hierarchical structure and encapsulation encoding of the video codec specification spanning over at least one hierarchy selected from a group of:
a video layer level;
a video sequence level;
a video coded frame level;
a video coded slice level; and
a video coded slice segment level.
9. The apparatus of claim 8 , wherein the at least one processor is configured to cause the apparatus to model a graphical directed information flow of the processing of a NAL unit's hierarchical structure and encapsulation encoding towards the determination of the information-to-importance value for each of the plurality of NAL units based at least on an information-to-importance functional kernel and a recursive accumulation of information flow across directed edges of the graphical directed information flow model formed of the plurality of NAL units as nodes.
10. The apparatus of claim 9 , wherein the information-to-importance functional kernel is defined for any level depth of the graphical directed information flow model processing as a functional parameter of a subset of an extracted feature set associated with a graph node i on the level as .
11. The apparatus of claim 10 , wherein the recursive accumulation of information flow across the directed edges of the graphical directed information flow model formed of NAL unit nodes is limited to a weighted aggregation of intra-level
directed edges and a direct lower level inter-level directed edges such that for a node i at level , wherein an accumulated importance value is obtained as
= +Σ j≠i +Σ k ,
wherein a weight of the aggregation between the node and a node j≠i at level is a scalar , and a weight of the aggregation between node and a node k at level
+1 is a scalar .
12. The apparatus of claim 11 , wherein the at least one processor is configured to cause the apparatus to limit the information-to-importance functional kernel
to a model-common information-to-importance functional kernel
.
13. The apparatus of claim 9 , wherein the information-to-importance functional kernel is a scalar constant for graphical directed information flow model nodes and levels.
14. The apparatus of claim 9 , wherein the at least one processor is configured to cause the apparatus to determine an importance value of the information-to-importance functional kernel for the video coded slice or slice segment level inversely proportional to a video compression gain relative to an uncompressed video picture resolution content under constant quality encodings such that frames that are compressed above a predefined threshold have low importance values and frames that are compressed below the predefined threshold have high importance values.
15. The apparatus of claim 9 , wherein the at least one processor is configured to cause the apparatus to determine an importance value of the information-to-importance functional kernel for the video coded slice or slice segment level based on at least one selected from a group comprising:
a video coded sequence of pictures;
a video frame type;
a video slice type; and
a video slice reference list.
16. The apparatus of claim 9 , wherein the at least one processor is configured to cause the apparatus to simplify a full connectivity of a plurality of video coding layers of at least one of spatial and temporal layer type to ignore graphical directed information flow model connections between layers at a lowest level and decoupling and offsetting the plurality of video coding layers by a constant scalar to determine their importance levels and their enclosed NAL units' importance levels.
17. The apparatus of claim 9 , wherein the at least one processor is configured to cause the apparatus to optimize a plurality of information-to-importance functional kernels and associated graphical directed information flow weights using statistical learning based on a generic set of training data of video coded sequences to at least one of minimize expected metric and maximize expected metric of visual rendering quality based on the plurality of feature sets associated with each of the plurality of NAL units of the video coded stream.
18. The apparatus of claim 1 , wherein the at least one processor is configured to cause the apparatus to signal the information-to-importance values of the plurality of NAL units to at least one of a radio access network (“RAN”) and a user equipment (“UE”) for optimization of transmission, retransmission, and reception procedures associated with the video coded stream over at least one of a limited capacity and varying capacity wireless link, the signaling performed using at least one indication selected from a group of a radio resource control (“RRC”) semi-static indication, a downlink control information (“DCI”) indication, and a dynamic header information indication, the optimization of transmission, retransmission, and reception procedures comprising at least one of packet scheduling, radio power control, radio frequency and resource allocation, physical layer modulation and coding scheme selection, hybrid automatic repeat request retransmissions preemptions, and multi-antenna and multiple transmission reception point transmissions, beamforming, and spatial multiplexing procedures.
19. A method of a network device, the method comprising:
detecting a plurality of video coded network abstraction layer (“NAL”) units of a video coded stream according to defined syntax elements of a determined video codec specification;
extracting semantic information associated with the plurality of NAL units according to a syntax elements and semantic knowledge base defined within the determined video codec specification;
combining the extracted semantic information associated with the plurality of NAL units to form a plurality of feature sets comprising a plurality of NAL unit type parameters, the plurality of feature sets correspondingly synchronized with the plurality of NAL units that enclose the extracted semantic information;
determining an information-to-importance value for each of the plurality of NAL units based on a NAL unit type, a NAL unit size, and a hierarchical encoding of each of the plurality of feature sets without performing video decoding of the video coded stream; and
indicating the determined information-to-importance value for each of the plurality of NAL units of the determined video codec specification to a video coded traffic-aware transceiver for scheduling video traffic based on the indicated information-to-importance value.
20. A remote network device apparatus, the apparatus comprising:
at least one memory; and
at least one processor coupled with the at least one memory and configured to cause the apparatus to:
encode and compress an uncompressed video sequence to a video coded stream formed of a plurality of network abstraction layer (“NAL”) units using a selected video codec specification;
extract semantic information associated with the plurality of NAL units according to a syntax elements and semantic knowledge base defined within the selected video codec specification;
combine the extracted semantic information associated with the plurality of NAL units to form a plurality of feature sets comprising a plurality of NAL unit type parameters, the plurality of feature sets correspondingly synchronized with the plurality of NAL units that enclose the extracted semantic information;
determine an information-to-importance value for each of the plurality of NAL units based on a NAL unit type, a NAL unit size, and a hierarchical encoding of each of the plurality of feature sets without performing video decoding of the video coded stream;
annotate the plurality of NAL units with the information-to-importance values for forming a plurality of application data units (“ADUs”) of the video coded stream for packet-switched communication networks;
signal the information-to-importance values for the plurality of ADUs and the plurality of NAL units;
process video coded traffic awareness information to determine optimized radio scheduling and control procedures for transmit and receive operations for the video coded stream ADUs; and
transmit the video coded stream ADUs to a radio access network (“RAN”) based on video coded traffic-aware optimizations.