Configurable positions for auxiliary information input into a picture data processing neural network
Methods and apparatuses for processing of picture data or picture feature data using a neural network with two or more layers are provided. The present disclosure may be applied in the field of artificial intelligence (AI)-based video or picture compression technologies, and in particular, to the field of neural network-based video compression technologies. According to the present disclosure, position within the neural network, at which auxiliary information may be entered for processing is selectable based on a gathering condition. The gathering condition may assess whether some prerequisite is fulfilled. Advantages may include better performance in terms of rate and/or disclosure due to the effect of increased flexibility in neural network configurability.
1 . A method for processing picture feature data from a bitstream using a neural network comprising a plurality of neural network layers, the method comprising:
obtaining the picture feature data from the bitstream;
processing the picture feature data using the neural network, wherein for each of one or more preconfigured positions within the neural network, the processing the picture feature data comprises:
determining, based on a gathering condition, whether to gather auxiliary data for processing by one of the plurality of neural network layers at the preconfigured position,
in response to determining to gather the auxiliary data, processing the picture feature data with the neural network layer at the preconfigured position based on the auxiliary data.
2 . The method according to claim 1 , wherein after applying the gathering condition in determining to gather the auxiliary data, the auxiliary data is to be gathered for a single one of the one or more preconfigured positions.
3 . The method according to claim 1 , wherein after applying the gathering condition in determining to gather the auxiliary data, the auxiliary data is to be gathered for more than one of the preconfigured positions.
4 . The method according to claim 1 , wherein
there are more than one of the preconfigured positions;
the auxiliary data is scalable in size to match dimensions of an input channel processed by the layer at two or more of the preconfigured positions; and
after applying the gathering condition in determining to gather the auxiliary data, the auxiliary data is
i) gathered or
ii) gathered and scaled
for a single one of the preconfigured positions.
5 . The method according to claim 1 , wherein the gathering condition is based on a picture characteristic or a picture feature data characteristic obtained from the bitstream.
6 . The method according to claim 5 , wherein
the picture characteristic or the picture feature data characteristic includes resolution; and
the gathering condition includes a comparison of the resolution with a preconfigured resolution threshold.
7 . The method according to claim 5 , wherein
the picture is a video picture and the picture characteristic includes a picture type; and
the gathering condition includes determining whether the picture type is a temporally predicted picture type or spatially predicted picture type.
8 . The method according to claim 1 , further comprising:
obtaining from the bitstream an indication specifying for the one or more preconfigured positions whether or not to gather the auxiliary data,
the gathering condition for each of the one or more preconfigured positions is as follows:
based on the indication specifies for the preconfigured position that the auxiliary data being to be gathered, the determination is affirmative; and
based on the indication specifies for the preconfigured position that the auxiliary data being not to be gathered, the determination is negative.
9 . The method according to claim 1 , wherein the auxiliary data provides information about the picture feature data processed by the neural network to generate an output.
10 . The method according to claim 1 , wherein the auxiliary data includes prediction data which is a prediction of the picture or a prediction of picture feature data after processing by one or more of the layers of the neural network;
wherein the auxiliary data are a coupled pair of the prediction data and supplementary data to be combined with the prediction data; and
wherein the prediction data and the supplementary data have dimensions of data processed by layers at mutually different positions in the neural network.
11 . The method according to claim 1 , wherein
the neural network includes a sub-network for lossless decoding with at least one layer; and
the auxiliary data is input into the sub-network for lossless decoding.
12 . The method according to claim 1 , wherein the neural network is trained to perform at least one of still picture decoding, video picture decoding, still picture filtering, video picture filtering, and machine vision processing including object detection, object recognition or object classification.
13 . The method according to claim 1 , wherein the method is performed for each of a plurality of auxiliary data, including first auxiliary data and second auxiliary data,
wherein the first auxiliary data is associated with a first set of one or more preconfigured positions and the second auxiliary data is associated with a second set of one or more preconfigured positions; and
wherein the first set of one or more preconfigured positions and the second set of one or more preconfigured positions share at least one preconfigured position.
14 . The method according to claim 1 ,
wherein the neural network is trained to perform the processing of video pictures; and
wherein the determining whether to gather the auxiliary data for processing by the neural network layer at the preconfigured position is performed on every predetermined number of video pictures, wherein the predetermined number of video pictures is one or more.
15 . A method for processing a picture with a neural network comprising a plurality of neural network layers to generate a bitstream, the method comprising:
processing the picture with the neural network, wherein for each of one or more preconfigured positions within the neural network, the processing the picture comprises:
determining, based on a gathering condition, whether to gather auxiliary data for processing by a layer at the preconfigured position,
in response to determining to gather the auxiliary data, processing the picture with the layer at the preconfigured position based on the auxiliary data; and
inserting into the bitstream data obtained through processing the picture by the neural network.
16 . The method according to claim 15 , wherein after applying the gathering condition in determining to gather the auxiliary data, the auxiliary data is to be gathered for a single one of the one or more preconfigured positions.
17 . The method according to claim 15 , wherein after applying the gathering condition in determining to gather the auxiliary data, the auxiliary data is to be gathered for more than one of the preconfigured positions.
18 . The method according to claim 15 , wherein
there are more than one of the preconfigured positions;
the auxiliary data is scalable in size to match dimensions of an input channel processed by the layer at two or more of the preconfigured positions; and
after applying the gathering condition in determining to gather the auxiliary data, the auxiliary data is
i) gathered or
ii) gathered and scaled
for a single one of the preconfigured positions.
19 . The method according to claim 15 , wherein the gathering condition is based on a picture characteristic or a picture feature data characteristic which is included into the bitstream.
20 . The method according to claim 19 , wherein
the picture characteristic or the picture feature data characteristic includes resolution; and
the gathering condition includes a comparison of the resolution with a preconfigured resolution threshold.
21 . The method according to claim 19 , wherein
the picture is a video picture and the picture characteristic includes a picture type; and
the gathering condition includes determining whether the picture type is a temporally predicted picture type or spatially predicted picture type.
22 . The method according to claim 15 , further comprising:
generating the indication specifying for the one or more preconfigured positions whether to gather the auxiliary data;
including the indication into the bitstream; and
selecting for the one or more preconfigured positions whether to gather the auxiliary data based on an optimization of a cost function including at least one of rate, distortion, accuracy, or complexity.
23 . A non-transitory computer-readable medium including code instructions which upon execution by one or more processors cause the one or more processors to perform the method according to claim 1 .
24 . An apparatus for processing picture feature data from a bitstream using a neural network comprising a plurality of neural network layers, the apparatus comprising:
a processing circuitry configured to:
obtain the picture feature data from the bitstream;
process the picture feature data using the neural network, wherein for each of one or more preconfigured positions within the neural network, the processing the picture feature data comprises:
determine, based on a gathering condition, whether to gather auxiliary data for processing by one of the plurality of neural network layers at the preconfigured position,
in response to determining to gather the auxiliary data, process the picture feature data with the neural network layer at the preconfigured position based on the auxiliary data.
25 . An apparatus for processing a picture with a neural network comprising a plurality of neural network layers to generate a bitstream, the apparatus comprising:
a processing circuitry configured to:
process the picture with the neural network, wherein for each of one or more preconfigured positions within the neural network, the processing the picture comprises:
determine, based on a gathering condition, whether to gather auxiliary data for processing by a layer at the preconfigured position,
in response to determining to gather the auxiliary data, process the picture with the layer at the preconfigured position based on the auxiliary data; and
insert into the bitstream data obtained through processing the picture by the neural network.