Apparatus and method with video processing using neural network
An apparatus with video processing includes: one or more processors configured to: generate a syntax element processable by a target standard codec by inputting a quantization parameter, a pre-decoded reference image, and a plurality of frames comprised in a video to a neural network and compressing the plurality of frames, and generate a bitstream by performing entropy encoding on the syntax element.
1 . An apparatus with video processing, the apparatus comprising:
one or more processors configured to:
generate syntax elements processable by a target decoder of a target standard codec by selecting one of outputs generated by inputting a quantization parameter, a pre-decoded reference image, and a plurality of frames comprised in a video to neural networks, and
generate a bitstream by performing entropy encoding on the syntax elements.
2 . The apparatus of claim 1 , wherein the syntax elements comprise a coding unit (CU) partition, a prediction unit (PU) partition, a PU prediction mode, and a transform unit (TU) partition.
3 . The apparatus of claim 1 , wherein the pre-decoded reference image is decoded at a time point before a time point when input frame is encoded.
4 . The apparatus of claim 1 , wherein, for the generating of the pre-decoded reference image, the one or more processors are further configured to:
generate decoded syntax elements by performing entropy decoding on the bitstream, and
generate the pre-decoded reference image by decompressing the decoded syntax elements.
5 . The apparatus of claim 4 , wherein the decoded syntax elements are decodable by the target decoder comprising a decoder of a standard codec.
6 . The apparatus of claim 1 , wherein the neural networks comprise a first neural network and a second neural network.
7 . The apparatus of claim 6 , wherein, for the generating of the bitstream, the one or more processors are further configured to:
select one from an output of the first neural network and an output of the second neural network, and
perform entropy encoding on the selected output.
8 . The apparatus of claim 6 , wherein, for the generating of the syntax elements, the one or more processors are further configured to perform either one or both of:
intra-prediction through the first neural network; and
inter-prediction through the second neural network.
9 . The apparatus of claim 7 , wherein, for the generating of the syntax elements, the one or more processors are further configured to:
partition the plurality of frames into a plurality of blocks through the first neural network, and
perform motion estimation and compensation by inputting the plurality of blocks to the second neural network.
10 . The apparatus of claim 1 , wherein, for the generating of the syntax elements, the one or more processors are further configured to adjust the quantization parameter based on a shape of adaptive instance normalization of a layer constituting the neural networks and a product or sum of features at an arbitrary level.
11 . A processor-implemented method with video processing, the method comprising:
generating syntax elements processable by a target decoder of a target standard codec by selecting one of outputs generated by inputting a quantization parameter, a pre-decoded reference image, and a plurality of frames comprised in a video to neural networks; and
generating a bitstream by performing entropy encoding on the syntax elements.
12 . The method of claim 11 , wherein the syntax elements comprise a coding unit (CU) partition, a prediction unit (PU) partition, a PU prediction mode, and a transform unit (TU) partition.
13 . The method of claim 11 , wherein the pre-decoded reference image is decoded at a time point before a time point when input frame is encoded.
14 . The method of claim 11 , wherein the generating of the pre-decoded reference image comprises:
generating decoded syntax elements by performing entropy decoding on the bitstream; and
generating the pre-decoded reference image by decompressing the decoded syntax elements.
15 . The method of claim 14 , wherein the decoded syntax elements are decodable by the target decoder comprising a decoder of a standard codec.
16 . The method of claim 11 , wherein the neural networks comprise a first neural network and a second neural network.
17 . The method of claim 16 , wherein the generating of the bitstream comprises:
selecting one from an output of the first neural network and an output of the second neural network; and
performing entropy encoding on the selected output.
18 . The method of claim 16 , wherein the generating of the syntax elements comprises either one or both of:
performing intra-prediction through the first neural network; and
performing inter-prediction through the second neural network.
19 . The method of claim 17 , wherein the generating of the syntax elements comprise:
partitioning the plurality of frames into a plurality of blocks through the first neural network; and
performing motion estimation and compensation by inputting the plurality of blocks to the second neural network.
20 . The method of claim 11 , wherein the generating of the syntax elements comprises adjusting the quantization parameter based on a shape of adaptive instance normalization of a layer constituting the neural networks and a product or sum of features at an arbitrary level.