Feature map encoding and decoding based on presence indicator
The present disclosure relates to efficient signaling of feature map information for a system employing a neural network. In particular, at the decoder side, a presence indicator is obtained based on information parsed from a bitstream. Based on the value of the obtained presence indicator, further data related to a feature map region are parsed or the parsing is bypassed. The presence indicator may be, for instance, a region presence indicator indicating whether feature map data is included in the bitstream or may be a side information presence indicator indicating whether a side information related to the feature map data is included in the bitstream. Similarly, an encoding method, as well as encoding and decoding devices, are provided. Accordingly, feature map data may be processed more efficiently, by reducing decoding complexity, and the amount of transmitted data can be reduced by applying the bypassing.
1 . A method for decoding, based on a bitstream, a feature map for processing by a neural network, the method comprising:
obtaining, for a region of the feature map, a region presence indicator based on information from the bitstream; and
decoding the region, wherein decoding the region includes:
parsing data from the bitstream for decoding the region based on the region presence indicator having a first value, or
bypassing parsing data from the bitstream for decoding the region based on the region presence indicator having a second value.
2 . The method according to claim 1 , wherein, based on the region presence indicator having a second value, the decoding the region further includes setting of the region according to a predetermined rule.
3 . The method according to claim 2 , wherein the predetermined rule specifies the setting of the features of the region to a constant.
4 . The method according to claim 3 , wherein the constant is zero.
5 . The method according to claim 3 , further comprising decoding the constant from the bitstream.
6 . The method according to claim 1 , wherein the bitstream includes the region presence indicator.
7 . The method according to claim 1 , further comprising obtaining side information from the bitstream,
wherein the obtaining the region presence indicator based on information from the bitstream comprises obtaining the region presence indicator based on the side information.
8 . The method according to claim 7 , further comprising:
obtaining a side information presence indicator from the bitstream; and
parsing the side information from the bitstream based on the side information presence indicator having a third value or bypassing parsing the side information from the bitstream based on the side information presence indicator having a fourth value,
wherein the side information includes the region presence indicator and/or information for being processed by a neural network to obtain an estimated probability model for use in an entropy decoding of the region.
9 . The method according to claim 8 , further comprising setting, based on the side information presence indicator having the fourth value, the side information to a predetermined side information value.
10 . The method according to claim 1 , wherein the region presence indicator is a flag capable of taking one of only two values formed by the first value and the second value.
11 . The method according to claim 1 , wherein the region is a channel of the feature map.
12 . The method according to claim 11 , further comprising:
obtaining a significance order that indicates significance of a plurality of channels of the feature map,
obtaining a last significant channel indicator, and
obtaining the region presence indicator based on the last significant channel indicator.
13 . The method according to claim 12 , wherein the last significant channel indicator corresponds to a quality indicator decoded from the bitstream and indicating quality of the coded feature map resulting from compression of the region of the feature map.
14 . The method according to claim 12 , wherein the last significant channel indicator corresponds to an index of the last significant channel within the significance order.
15 . The method according to claim 12 , wherein the obtaining the significance order comprises decoding an indication of the significance order from the bitstream.
16 . The method according to claim 12 , wherein the obtaining the significance order comprises deriving the significance order based on previously decoded information regarding source data from which the feature map has been generated.
17 . The method according to claim 12 , wherein the obtaining the significance order comprises deriving the significance order based on previously decoded information regarding a type of source data from which the feature map has been generated.
18 . The method according to claim 12 , further comprising decoding, from the bitstream, channels sorted within the bitstream according to the significance order from a most significant channel to a least significant channel.
19 . The method according to claim 1 , further comprising:
decoding, from the bitstream, region splitting information indicating a splitting of the region of the feature map into units, and
based on the region splitting information, obtaining a unit presence indicator that indicates whether or not feature map data is to be parsed from the bitstream for decoding a unit of the region.
20 . The method according to claim 19 , wherein the region splitting information for the region includes a flag indicating whether or not the bitstream comprises unit information specifying dimensions and/or position of the units of the region, the method further comprising:
obtaining, based on information from the bitstream, a respective unit presence indicator for each respective unit of the region; or
based on a value of the respective unit presence indicator for a respective unit, parsing or not parsing from the bitstream feature map data for the respective unit.
21 . The method according to claim 20 , wherein the unit information specifies hierarchic splitting of the region including at least one of quad-tree, binary-tree, ternary-tree, or triangular splitting.
22 . The method according to claim 19 , wherein the bitstream includes the unit presence indicator.
23 . The method according to claim 1 , wherein the decoding of the region includes:
extracting, from the bitstream, a last significant coefficient indicator that indicates a position of a last coefficient among coefficients of the region,
decoding significant coefficients of the region from the bitstream,
setting the coefficients following the last significant coefficient indicator according to a predefined rule, and
obtaining feature data of the region by inverse transformation of the coefficients of the region.
24 . The method according to claim 23 , wherein the inverse transformation is one of: an inverse discrete cosine transformation, an inverse discrete sine transformation, an inverse transformation obtained by modifying the inverse discrete cosine transformation or inverse discrete sine transformation, or a convolutional neural network transform.
25 . The method according to claim 1 , further comprising decoding from the bitstream a side information presence flag that indicates whether or not the bitstream includes any side information for the feature map, the side information including information to be processed by a neural network to obtain an estimated probability model for use in an entropy decoding of the feature map.
26 . The method according to claim 1 , wherein the obtaining the region presence indicator includes decoding by a context adaptive entropy decoder.
27 . A method for computer vision, the method comprising:
performing the method according to claim 1 for decoding, from the bitstream, the feature map for processing by the neural network; and
performing a computer vision task comprising processing the decoded feature map with the neural network.
28 . A computer program stored on a non-transitory processor readable medium comprising processor executable code that, when executed by one or more processors, causes the one or more processors to perform the method according to claim 1 .
29 . A device for decoding a feature map for processing by a neural network based on a bitstream, the device comprising a processing circuitry configured to perform the method according to claim 1 .
30 . The method according to claim 1 , wherein the region of the feature map corresponds to a spatially contiguous subset of an image.
31 . The method of claim 30 , wherein the value of the region presence indicator is determined based on an importance of the region of the feature map to a reconstruction quality of the image.
32 . The method of claim 31 , wherein the region presence indicator has the first value when the region of the feature map is non-flat, and
wherein the region presence indicator has the second value when the region of the feature map is flat.
33 . The method of claim 30 , wherein the value of the region presence indicator is determined based on a number of bits required to transit the region of the feature map.
34 . The method according to claim 30 , wherein the value of the region presence indicator is determined by rate-distortion optimization (RDO).
35 . The method according to claim 1 , wherein the feature map is an output of one or more convolutional layers of a convolutional neural network.
36 . A method for decoding an image, the method comprising:
performing the method according to claim 1 for decoding, from the bitstream, the feature map for processing by the neural network; and
processing the decoded feature map with the neural network to obtain the decoded image.
37 . The method according to claim 36 , wherein the feature map represents: coded image data and/or coded side information for decoding the image data.
38 . The method according to claim 36 , wherein the computer vision task is object detection, object classification, and/or object recognition.
39 . A method for encoding, into a bitstream, a feature map for processing by a neural network, the method comprising:
obtaining, for a region of the feature map, a region presence indicator; and
implementing, based on the obtained region presence indicator, a decision to perform:
encoding, based on the region presence indicator having a first value, the region of the feature map into the bitstream; or
bypassing encoding, based on the region presence indicator having a second value, of the region of the feature map into the bitstream.
40 . The method according to claim 39 , wherein the region presence indicator is indicated in the bitstream.
41 . The method according to claim 39 , wherein the deciding includes evaluating values of the features of the region.
42 . The method according to claim 39 , wherein implementing the decision is based on an impact of the region on a quality of a result of the neural network processing.
43 . The method according to claim 39 , wherein implementing the decision comprises:
determining, incrementally, a sum of bits required for transmission of the feature map, starting from bits of most significant regions and continuing to bits of regions with decreasing significance until the sum exceeds a pre-configured threshold,
encoding regions for which the sum did not exceed the pre-configured threshold and the region presence indicator having the first value for the encoded regions, and
encoding the region presence indicator having the second value for non encoded regions.
44 . A device for encoding a feature map for processing by a neural network into a bitstream, the device comprising a processing circuitry configured to perform the method according to claim 39 .
45 . A computer program stored on a non-transitory processor readable medium comprising processor executable code that, when executed by one or more processors, causes the one or more processors to perform the method according to claim 39 .
46 . A device for decoding, based on a bitstream, a feature map for processing by a neural network, the device comprising:
processing circuitry, the processing circuitry comprising:
a region presence indicator obtaining module configured to obtain, for a region of the feature map, a region presence indicator based on information from the bitstream,
a decoding module configured to decode the region by:
parsing, based on the region presence indicator having a first value, data from the bitstream for decoding the region, and
bypassing parsing, based on the region presence indicator having a second value, of data from the bitstream for decoding the region.
47 . A device for encoding, into a bitstream, a feature map for processing by a neural network, the device comprising:
processing circuitry, the processing circuitry comprising:
a feature map region presence indicator obtaining module configured to obtain the feature map region presence indicator,
an encoding control module configured to implement a decision, based on the obtained feature map region presence indicator, to:
encode, based on the feature map region presence indicator having a first value, the region of the feature map into the bitstream;
bypass encoding, based on the feature map region presence indicator having a second value, of the region of the feature map into the bitstream.