Channel dynamic range adjustment method for feature tensor compression in split inference
Systems, methods, and instrumentalities are disclosed for performing channel dynamic range adjustment and/or channel dynamic range restoration associated with feature tensor compression in split inference. Range adjustment associated with a channel associated with a feature tensor may be performed. Range adjustment may include a shift and/or scale operation. Range adjustment model(s) (e.g., shift and/or scale parameters) may be determined, for example, based on optimization during encoding. The range adjustment model(s) may be used to adapt distributions between channels for a tensor packed frame. The range adjustment model(s) may be channel-level parameters (e.g., a range adjustment model may be associated with a specific channel; a range adjustment model may be per-channel).
1 . A video encoding device, the video encoding device comprising:
a processor configured to:
obtain a feature tensor associated with a video;
perform a channel dynamic range adjustment, wherein the channel dynamic range adjustment comprises applying a first learned affine range adjustment model to a first channel associated with the feature tensor and applying a second learned affine range adjustment model to a second channel associated with the feature tensor, and wherein an adjusted feature tensor is determined based on the performed channel dynamic range adjustment; and
encode the video based on the adjusted feature tensor.
2 . The video encoding device of claim 1 , wherein the first range adjustment model is associated with a first scale parameter and a first shift parameter, wherein the second range adjustment model is associated with a second scale parameter and a second shift parameter.
3 . The video encoding device of claim 2 , wherein first scale parameter and the first shift parameter are determined using the first learned affine range adjustment model associated with the first channel, and wherein the second scale parameter and the second shift parameter are determined using the second learned affine range adjustment model associated with the second channel.
4 . The video encoding device of claim 1 , wherein the feature tensor is a normalized feature tensor, wherein the normalized feature tensor is obtained based on a minimum value and a maximum value associated with the feature tensor, and wherein based on a determination to use channel dynamic range adjustment on the feature tensor, the minimum value is zero.
5 . The video encoding device of claim 1 , wherein processor is further configured to send a channel dynamic range adjustment indication, wherein the channel dynamic range adjustment indication indicates to bypass channel dynamic range restoration associated with encoded adjusted feature tensor.
6 . The video encoding device of claim 1 , wherein processor is further configured to send a channel dynamic range adjustment indication, wherein the channel dynamic range adjustment indication indicates to use at least one of the first learned affine range adjustment model or the second learned affine range adjustment model on the encoded adjusted feature tensor.
7 . The video encoding device of claim 1 , wherein processor is further configured to send a channel dynamic range adjustment indication, wherein the channel dynamic range adjustment indication indicates to use at least one of a third learned affine range adjustment model or a fourth learned affine range adjustment model on the encoded adjusted feature tensor, wherein the third learned affine range adjustment model is a previously decoded range adjustment model, and wherein the fourth learned affine range adjustment model is a previously decoded range adjustment model.
8 . The video encoding device of claim 1 , wherein processor is further configured to send a channel dynamic range adjustment indication, wherein the channel dynamic range adjustment indication indicates to use a pre-configured range adjustment model on the encoded adjusted feature tensor.
9 . The video encoding device of claim 1 , wherein processor is further configured to send a channel dynamic range adjustment indication, wherein the channel dynamic range adjustment indication indicates the first learned affine range adjustment model and the second learned affine range adjustment model, and wherein the first learned affine range adjustment model and second learned affine range adjustment model are indicated in a floating point precision.
10 . The video encoding device of claim 1 , wherein first learned affine range adjustment model and the second learned affine range adjustment model are at least one of a linear model or a logarithmic model.
11 . The video encoding device of claim 1 , wherein the processor is further configured to:
determine whether to use channel dynamic range adjustment on the feature tensor, wherein the channel dynamic range adjustment is performed based on a determination to use channel dynamic range adjustment on the feature tensor.
12 . A video decoding device, the video decoding device comprising:
a processor configured to:
obtain a feature tensor associated with a video;
perform channel dynamic range restoration on the feature tensor, wherein the channel dynamic range restoration comprises applying a first learned affine range adjustment model to a first channel associated with the feature tensor and applying a second learned affine range adjustment model to a second channel associated with the feature tensor, and wherein an adjusted feature tensor is determined based on the performed channel dynamic range restoration;
determine reconstructed features based on the performed channel dynamic range restoration; and
decode the video based on the determined reconstructed features.
13 . The video decoding device of claim 12 , wherein the processor is further configured to:
obtain a channel dynamic range adjustment indication, wherein the channel dynamic range adjustment indication indicates to use channel dynamic range restoration on the tensor frame, wherein the channel dynamic range restoration is performed based on the indication to use channel dynamic range restoration on the tensor frame, wherein the channel dynamic range adjustment indication indicates to use the first learned affine range adjustment model and the second learned affine range adjustment model, wherein the first learned affine range adjustment model and the second learned affine range adjustment model are indicated in a floating point precision.
14 . The video decoding device of claim 11 , wherein the processor is further configured to obtain a channel dynamic range adjustment indication, wherein the channel dynamic range adjustment indication indicates at least one of,
to bypass channel dynamic range restoration on the unpacked feature tensor,
the first learned affine range adjustment model, the second learned affine range adjustment model, and to use the first learned affine range adjustment model and the second learned affine range adjustment model for channel dynamic range restoration associated with the feature tensor;
to use at least one previously decoded range adjustment model for channel dynamic range restoration associated with the feature tensor, or
to use at least one pre-configured range adjustment model for channel dynamic range restoration associated with the feature tensor.
15 . A video encoding method, the method comprising:
obtaining a feature tensor associated with a video;
performing a channel dynamic range adjustment, wherein the channel dynamic range adjustment comprises applying a first learned affine range adjustment model to a first channel associated with the feature tensor and applying a second learned affine range adjustment model to a second channel associated with the feature tensor, and wherein an adjusted feature tensor is determined based on the performed channel dynamic range adjustment; and
encoding the video based on the adjusted feature tensor.
16 . The video encoding method of claim 15 , wherein the first learned affine range adjustment model is associated with a first scale parameter and a first shift parameter, wherein the second range learned affine adjustment model is associated with a second scale parameter and a second shift parameter.
17 . The video encoding method of claim 16 , wherein first scale parameter and the first shift parameter are determined using the first learned affine range adjustment model associated with the first channel, and wherein the second scale parameter and the second shift parameter are determined using the second learned affine range adjustment model associated with the second channel.
18 . The video encoding method of claim 15 , wherein the feature tensor is a normalized feature tensor, wherein the normalized feature tensor is obtained based on a minimum value and a maximum value associated with the feature tensor, and wherein based on a determination to use channel dynamic range adjustment on the feature tensor, the minimum value is zero.
19 . The video encoding method of claim 15 , wherein the method further comprises:
sending a channel dynamic range adjustment indication, wherein the channel dynamic range adjustment indication indicates to perform channel dynamic range adjustment and indicates at least one of:
to bypass channel dynamic range restoration on the unpacked feature tensor,
the first learned affine range adjustment model, the second learned affine range adjustment model, and to use the first learned affine range adjustment model and the second learned affine range adjustment model for channel dynamic range restoration associated with the feature tensor;
to use at least one previously decoded range adjustment model for channel dynamic range restoration associated with the feature tensor, or
to use at least one pre-configured range adjustment model for channel dynamic range restoration associated with the feature tensor.
20 . The video encoding method of claim 15 , wherein the method further comprises:
sending a channel dynamic range adjustment indication, wherein the channel dynamic range adjustment indication indicates the first learned affine range adjustment model and the second learned affine range adjustment model, and wherein the first learned affine range adjustment model and second learned affine range adjustment model are indicated in a floating point precision.