Apparatus and method with video processing using neural network
An apparatus with video processing includes one or more processors configured to generate a syntax element processable by a target standard codec by inputting a quantization parameter, a pre-decoded reference image, and a plurality of frames comprised in a video to a neural network and compressing the plurality of frames, and generate a bitstream by performing entropy encoding on the syntax element.
1 . An apparatus with video processing, the apparatus comprising:
one or more processors configured to:
generate, based on decoder information on a decoder among a plurality of decoders, syntax elements processable by a target decoder of a target standard codec by inputting a quantization parameter, a pre-decoded reference image, and a plurality of frames comprised in a video to neural networks and compressing the plurality of frames, and
generate a bitstream by performing entropy encoding on the syntax elements.
2 . The apparatus of claim 1 , wherein the syntax elements comprises a coding unit (CU) partition, a prediction unit (PU) partition, a PU prediction mode, and a transform unit (TU) partition.
3 . The apparatus of claim 1 , wherein the pre-decoded reference image is decoded at a time point before a time point when input frame is encoded.
4 . The apparatus of claim 1 , wherein, for the generating of the pre-decoded reference image, the one or more processors are further configured to:
generate decoded syntax elements by performing entropy decoding on the bitstream, and
generate the pre-decoded reference image by decompressing the decoded syntax elements.
5 . The apparatus of claim 4 , wherein
the decoder is a target decoder comprising a decoder of a standard codec, and
the decoded syntax elements are decodable by the target decoder.
6 . The apparatus of claim 1 , wherein the neural networks comprise a first neural network and a second neural network.
7 . The apparatus of claim 6 , wherein, for the generating of the bitstream, the one or more processors are further configured to:
select one from an output of the first neural network and an output of the second neural network, and
perform entropy encoding on the selected output.
8 . The apparatus of claim 6 , wherein, for the generating of the syntax elements, the one or more processors are further configured to perform either one or both of:
intra-prediction through the first neural network; and
inter-prediction through the second neural network.
9 . The apparatus of claim 7 , wherein, for the generating of the syntax elements, the one or more processors are further configured to:
partition the plurality of frames into a plurality of blocks through the first neural network, and
perform motion estimation and compensation by inputting the plurality of blocks to the second neural network.
10 . The apparatus of claim 1 , wherein, for the generating of the syntax elements, the one or more processors are further configured to adjust the quantization parameter based on a shape of adaptive instance normalization of a layer constituting the neural networks and a product or sum of features at an arbitrary level.
11 . The apparatus of claim 1 , wherein, for the generating of the syntax elements, the one or more processors are further configured to:
receive the decoder information on the decoder to perform decoding among a plurality of decoders, and
generate the syntax elements by considering the decoder information.
12 . A processor-implemented method with video processing, the method comprising:
generating, based on decoder information on a decoder among a plurality of decoders, syntax elements processable by a target decoder of a target standard codec by inputting a quantization parameter, a pre-decoded reference image, and a plurality of frames comprised in a video to neural networks and compressing the plurality of frames; and
generating a bitstream by performing entropy encoding on the syntax elements.
13 . The method of claim 12 , wherein the syntax elements comprises a coding unit (CU) partition, a prediction unit (PU) partition, a PU prediction mode, and a transform unit (TU) partition.
14 . The method of claim 12 , wherein the pre-decoded reference image is decoded at a time point before a time point when input frame is encoded.
15 . The method of claim 12 , wherein the generating of the pre-decoded reference image comprises:
generating decoded syntax elements by performing entropy decoding on the bitstream; and
generating the pre-decoded reference image by decompressing the decoded syntax elements.
16 . The method of claim 15 , wherein
the decoder is a target decoder comprising a decoder of a standard codec, and
the decoded syntax elements are decodable by the target decoder.
17 . The method of claim 12 , wherein the neural networks comprise a first neural network and a second neural network.
18 . The method of claim 17 , wherein the generating of the bitstream comprises:
selecting one from an output of the first neural network and an output of the second neural network; and
performing entropy encoding on the selected output.
19 . The method of claim 17 , wherein the generating of the syntax elements comprises either one or both of:
performing intra-prediction through the first neural network; and
performing inter-prediction through the second neural network.
20 . The method of claim 18 , wherein the generating of the syntax elements comprises:
partitioning the plurality of frames into a plurality of blocks through the first neural network; and
performing motion estimation and compensation by inputting the plurality of blocks to the second neural network.
21 . The method of claim 12 , wherein the generating of the syntax elements comprises adjusting the quantization parameter based on a shape of adaptive instance normalization of a layer constituting the neural networks and a product or sum of features at an arbitrary level.
22 . The method of claim 12 , wherein the generating of the syntax elements comprises:
receiving the decoder information on the decoder to perform decoding among a plurality of decoders; and
generating the syntax element by considering the decoder information.