Methods and apparatus for kernel tensor and tree partition based neural network compression framework
A method of encoding or decoding a video comprising a current picture, a first reference picture, and a weight tensor associated with a trained neural network (NN) model are provided. The method includes generating any number of kernel tensors, input channels and output channels associated with the weight tensor, each kernel tensor being associated with any of: a layer type, an input signal type, and a tree partition type, and each kernel tensor including weight coefficients, generating, for each of the any number of kernel tensors, tree partitions for any of a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), and a transform unit (TU) according to respective tree partition types associated with each of the any number of kernel tensors, and generating a compressed representation of the trained NN model by compressing and coding the any number of kernel tensors.
1 . A method of decoding a trained neural network (NN) (TNN), the method comprising:
determining a first weight tensor (WT) associated with the TNN and a second WT associated with the TNN;
determining information associated with the TNN, wherein the information associated with the TNN indicates a layer type associated with a convolution NN (CNN);
generating an image based at least on an arrangement of the first WT and the second WT of the TNN;
generating a CNN layer based on the arrangement of the first WT and the second WT of the TNN, wherein the CNN layer includes a plurality of kernel tensors (KTs), wherein the plurality of the KTs includes a set of values of the first WT of the TNN and the second WT of the TNN, and wherein for each KT, an entirety of the each KT is used for generating tree partitions; and
outputting an NN model, wherein the NN model includes at least the generated CNN layer.
2 . The method of claim 1 , wherein, for the each KT, the entirety of the each KT is used for generating tree partitions for any of a prediction unit (PU) or a transform unit (TU).
3 . The method of claim 1 , further comprising using or selecting any of a coding syntax, a coding mode, or a coding method according to any of a dimension of a KT from the plurality of KTs and a size of the KT from the plurality of KTs.
4 . The method of claim 1 , wherein the layer type is any of: a convolutional layer type, a fully connected layer type, and a bias layer type.
5 . The method of claim 1 , wherein the method further comprises:
receiving a signal, wherein the signal is associated with one of a three-dimensional (3D) signal type associated with a video or a point cloud, a two-dimensional (2D) signal type associated with the image, or a one-dimensional (1D) signal type associated with audio, wherein the received signal is configured to be a picture and a number of reference pictures according to the arrangement of the received WTs of the TNN.
6 . The method of claim 1 , wherein the KTs are associated with a tree partition type (TPT), and wherein the TPT is one of a single tree type, a multi-tree type, or a mixed tree type.
7 . The method of claim 1 , wherein the plurality of KTs are divided into a plurality of sub-tenors, wherein each sub-tensor in the plurality of sub-tensors includes a respective number of weight coefficients.
8 . The method of claim 7 , further comprising compressing coding of the plurality of KTs by compressing and coding the plurality of sub-tensors associated with the KTs.
9 . The method of claim 8 , further comprising compressing and coding at least one of the KTs or sub-tensors by performing tree partitions on the at least one of the KTs or the sub-tensors and a quantization unit (QU).
10 . The method of claim 1 , wherein generating the CNN layer further comprises,
determining, for the plurality of KTs, at least one of: an input signal type of the each KT in the plurality of KTs, or a plurality of tree partition types (TPTs) of the each KT; and
decoding the plurality of KTs according to one of the input signal type of the each KT or the determined TPTs for each KT in the plurality of KTs, wherein the TPTs are one of a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), or a transform unit (TU).
11 . A wireless transmit receive unit (WTRU) comprising circuitry including a transmitter, a receiver, a processor and memory, configured to:
determining a first weight tensor (WT) associated with the TNN and a second WT associated with the TNN;
determine information associated with the TNN, wherein the information associated with the TNN indicates a layer type associated with a convolution NN (CNN);
generate an image based at least on an arrangement of the first WT and the second WT of the TNN;
generate a CNN layer based on the arrangement of the first WT and the second WT of the TNN, wherein the CNN layer includes a plurality of kernel tensors (KTs), wherein the plurality of the KTs includes a set of values of the first WT of the TNN and the second WT of the TNN, and wherein for each KT, an entirety of the each KT is used for generating tree partitions; and
output an NN model, wherein the NN model includes at least the generated CNN layer.
12 . The WTRU of claim 11 , wherein, for the each KT, the entirety of the each KT is used for generating tree partitions for any of a prediction unit (PU) or a transform unit (TU).
13 . The WTRU of claim 11 , configured to use or select any of a coding syntax, a coding mode, or a coding method according to any of a dimension of a KT from the plurality of KTs and a size of the KT from the plurality of KTs.
14 . The WTRU of claim 11 , wherein the layer type is any of: a convolutional layer type, a fully connected layer type, and a bias layer type.
15 . The WTRU of claim 11 , wherein the processor is further configured to:
receive a signal, wherein the signal is associated with one of a three-dimensional (3D) signal type associated with a video or a point cloud, a two-dimensional (2D) signal type associated with the image, or a one-dimensional (1D) signal type associated with audio, wherein the received signal is configured to be a picture and a number of reference pictures according to the arrangement of the received WTs of the TNN.
16 . The WTRU of claim 11 , wherein the KTs are associated with a tree partition type (TPT), and wherein the TPT is one of a single tree type, a multi-tree type, or a mixed tree type.
17 . The WTRU of claim 11 , wherein the plurality of KTs are divided into a plurality of sub-tenors, wherein each sub-tensor in the plurality of sub-tensors includes a respective number of weight coefficients.
18 . The WTRU of claim 17 , wherein the processor is further configured to compress coding of the plurality of KTs by compressing and coding the plurality of sub-tensors associated with the KTs.
19 . The WTRU of claim 18 , wherein the processor is further configured to compress and code at least one of the KTs or sub-tensors by performing tree partitions on the at least one of the KTs or the sub-tensors and a quantization unit (QU).
20 . The WTRU of claim 1 , wherein for the generation of the CNN layer, the processor is further configured to:
determine, for the plurality of KTs each KT, at least one of: an input signal type of the each KT in the plurality of KTs, or a plurality of tree partition types (TPTs) of the each KT; and
decode the plurality of KTs according to one of the input signal type of the each KT or the determined TPTs for each KT in the plurality of KTs, wherein the TPTs are one of a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), or a transform unit (TU).