Independent positioning of auxiliary information in neural network based picture processing
This application provides methods and apparatuses for processing of picture data or picture feature data using a neural network with two or more layers. The present disclosure may be applied in the field of artificial intelligence (AI)-based video or picture compression technologies, and in particular, to the field of neural network-based video compression technologies. According to some embodiments, two kinds of data are combined during the processing including processing by the neural network. The two kinds of data are obtained from different stages of processing by the network. Some of the advantages may include greater scalability and a more flexible design of the neural network architecture which may further lead to better encoding/decoding performance.
1 . A method for processing feature data of one or more pictures from a bitstream using a neural network which comprises two or more stages including a first stage and a second stage, the method comprising:
obtaining first data based on the bitstream,
processing the first data using the neural network including:
obtaining second data, which is based on data processed previously by the neural network, from the second stage of the neural network, and
using the first data together with the second data to generate input to the first stage of the neural network, wherein the first stage precedes the second stage in the feature data processing of the neural network; and
outputting the result of the processing,
wherein the first data is obtained in a third stage of the neural network preceding the first stage and the second stage or equal to the first stage in the feature data processing of the neural network.
2 . The method according to claim 1 , wherein the first data represents a prediction error and the second data represents a prediction.
3 . The method according to claim 2 , wherein the prediction is obtained by:
obtaining reference feature data which is feature data outputted by the neural network at said second stage,
obtaining, based on the bitstream, prediction information including motion information or spatial information related to the reference feature data,
generating the prediction based on the reference feature data and the prediction information.
4 . The method according to claim 3 , wherein
the prediction error is obtained by processing a current picture with the neural network;
the prediction information is motion information and the reference feature data is generated by the neural network processing of picture data representing picture preceding the current picture in decoding order.
5 . The method according to claim 4 , wherein the motion information includes motion vectors.
6 . The method according to claim 2 , wherein the using the first data together with the second data includes element-wise addition of the prediction or re-scaled prediction with the prediction error or re-scaled prediction error.
7 . The method according to claim 1 , wherein said second stage is the output of the neural network.
8 . The method according to claim 1 , wherein the first stage is the input of the neural network.
9 . The method according to claim 1 , wherein
the first stage is input of the neural network;
the first data is entropy encoded data;
the second stage is a stage different from the output stage of the neural network; and
the second data is probability model data related to the feature data of the second stage,
wherein the second data is probability model data for the entropy decoding of the first data.
10 . The method according to claim 1 , wherein
the position of the first stage and/or the second stage are configurable within the neural network, and
the method comprises configuring the position of the first stage and/or the second stage according to a gathering condition based on one or more picture coding parameters.
11 . The method according to claim 1 , further comprising
parsing from the bitstream an stage selection indicator specifying said first stage and/or said second stage,
wherein the position of the first stage and the second stage within the neural network is configured according to the parsed stage selection indicator.
12 . The method according to claim 1 , wherein:
the position of said first stage and/or said second stage are configurable within the neural network, and
the method comprises configuring the position of said first stage and/or said second stage according to a gathering condition based on one or more picture coding parameters.
13 . A computer program stored on a computer-readable, non-transitory medium and including code instructions which, when executed on one or more processors, cause the one or more processors to perform the method according to claim 1 .
14 . A method for processing at least one picture using a neural network, which comprises two or more stages including a first stage and a second stage, to generate a bitstream, the method comprising:
processing the at least one picture with the neural network including:
obtaining first data based on the at least one picture, and
obtaining second data at said second stage of the processing, the second data being based on data processed previously by the neural network;
using the first data together with the second data to generate input at said first stage of the neural network, wherein the first stage precedes the second stage in the feature data processing of the neural network; and
including into the bitstream feature data obtained by the processing,
wherein the first data is obtained in a third stage of the neural network preceding said first stage and said second stage or equal to said first stage in the feature data processing of the neural network.
15 . The method according to claim 14 , wherein the first data represents a prediction error and the second data represents a prediction.
16 . The method according to claim 15 , wherein the prediction is obtained by:
obtaining reference feature data which is feature data outputted by the neural network at said second stage,
obtaining, based on the at least one picture, prediction information including motion information or spatial information related to the reference feature data,
generating the prediction based on the reference feature data and the prediction information, and
inserting the obtained prediction information into the bitstream,
wherein the prediction error is obtained by processing a current picture with the neural network, and
wherein the prediction information is motion information and the reference feature data is generated by the neural network processing of picture data representing picture preceding the current picture in decoding order.
17 . The method according to claim 16 , wherein the motion information includes motion vectors.
18 . The method according to claim 15 , wherein the using the first data together with the second data includes element-wise subtraction of the prediction or re-scaled prediction from the first data or or re-scaled first data.
19 . The method according to any of claim 14 , wherein said second stage is the output of the decoding neural network representing reconstructed picture data.
20 . The method according to any of claim 14 , wherein said first stage is the output of the neural network.
21 . The method according to claim 14 , wherein
the first stage is output of the neural network;
the first data is processed data to be entropy encoded;
the second stage is a stage different from the input stage of the neural network; and
the second data is probability model data related to the feature data of the second stage,
wherein the second data is probability model data for the entropy encoding of the first data.
22 . An apparatus for processing feature data of one or more pictures from a bitstream using a neural network which comprises two or more stages including a first stage and a second stage, the apparatus comprising:
processing circuitry configured to:
obtain first data based on the bitstream;
process the first data using the neural network including:
obtaining second data, which is based on data processed previously by the neural network, from the second stage of the neural network, and
using the first data together with the second data to generate input to the first stage of the neural network, wherein the first stage precedes the second stage in the feature data processing of the neural network; and
output the result of the processing,
wherein the first data is obtained in a third stage of the neural network preceding the first stage and the second stage or equal to the first stage in the feature data processing of the neural network.
23 . An apparatus for processing at least one picture using a neural network, which comprises two or more stages including a first stage and a second stage, to generate a bitstream, the apparatus comprising:
processing circuitry configured to:
process the at least one picture with the neural network including:
obtaining first data based on the at least one picture, and
obtaining second data at said second stage of the processing, the second data being based on data processed previously by the neural network;
use the first data together with the second data to generate input at said first stage of the neural network, wherein the first stage precedes the second stage in the feature data processing of the neural network; and
include into the bitstream feature data obtained by the processing,
wherein the first data is obtained in a third stage of the neural network preceding said first stage and said second stage or equal to said first stage in the feature data processing of the neural network.