System, method, and computer program for content adaptive online training for multiple blocks based on certain patterns
Content-adaptive online training for end-to-end (E2E) neural image compression (NIC) using a neural network performed by at least one processor, is provided, including receiving an input image, to an E2E NIC framework, splitting the input image into a plurality of blocks, selecting a subset of blocks from the plurality of blocks, the subset of blocks sharing a same pattern, preprocessing a neural network of the E2E NIC framework, wherein the preprocessed neural network is applied to the selected subset of blocks, computing updated parameters using the preprocessed neural network, and generating an updated E2E NIC framework, based on the updated parameters.
1 . A method of content-adaptive online training for end-to-end (E2E) neural image compression (NIC) using a neural network performed by at least one processor, the method comprising:
receiving, by an E2E NIC framework, an input image comprising a plurality of blocks;
preprocessing a neural network of the E2E NIC framework using a preprocessed neural network based on the plurality of blocks;
computing updated bias parameters using a last layer of the preprocessed neural network, wherein the updated bias parameters comprise first updated bias parameters that are associated with a first subset of blocks from the plurality of blocks that share a same pattern, the first subset of blocks having one of a same RGB variance, a same YUV variance, and a same dominant color; and
generating an updated E2E NIC framework, based on the updated bias parameters.
2 . The method according to claim 1 , further comprising:
encoding the plurality of blocks and the updated bias parameters to generate a compressed representation of the plurality of blocks and a compressed representation of the updated bias parameters;
decoding the compressed representation of the updated bias parameters to generate decoded updated bias parameters;
updating the E2E NIC framework based on the decoded updated bias parameters; and
decoding the compressed representation of the plurality of blocks based on the updated E2E NIC framework to generate a reconstructed image.
3 . The method according to claim 2 , further comprising determining a distortion loss of the reconstructed image based on a consumption of the compressed representation of the plurality of blocks and the updated bias parameters, trade-off hyperparameter, and a distortion between block residuals of the compressed representation of the plurality of blocks and a block residual of the decoded compressed representations of the plurality of blocks.
4 . The method according to claim 1 , wherein the same pattern is determined based on an RGB variance of the plurality of blocks or a YUV variance of the plurality of blocks.
5 . The method according to claim 1 , wherein the updated bias parameters include a learning rate and a number of steps, and the learning rate and the number of steps are selected based on characteristics of the input image.
6 . The method according to claim 5 , wherein the characteristics of the input image are one of a RGB variance of the input image and an RD performance of the input image.
7 . The method according to claim 1 , wherein when preprocessing the neural network, the neural network is fine-tuned using the plurality of blocks.
8 . An apparatus for content-adaptive online training for end-to-end (E2E) neural image compression (NIC) using a neural network, the apparatus comprising:
at least one memory configured to store computer program code; and
at least one processor configured to read the computer program code and operate as instructed by the computer program code, the computer program code including:
receiving code configured to cause the at least one processor to receive, by an E2E NIC framework, an input image comprising a plurality of blocks;
preprocessing code configured to cause the at least one processor to preprocess a neural network of the E2E NIC framework using a preprocessed neural network based on the plurality of blocks;
computing code configured to cause the at least one processor to compute updated bias parameters using a last layer of the preprocessed neural network, wherein the updated bias parameters comprise first updated bias parameters that are associated with a first subset of blocks from the plurality of blocks that share a same pattern, the first subset of blocks having one of a same RGB variance, a same YUV variance, and a same dominant color; and
generating code configured to cause the at least one processor to generate an updated E2E NIC framework, based on the updated bias parameters.
9 . The apparatus of claim 8 , the computer program code further including:
encoding code configured to cause the at least one processor to encode the plurality of blocks and the updated bias parameters to generate a compressed representation of the plurality of blocks and a compressed representation of the updated bias parameters;
first decoding code configured to cause the at least one processor to decode the compressed representation of the updated bias parameters to generate decoded updated bias parameters;
updating code configured to cause the at least one processor to update the E2E NIC framework based on the decoded updated bias parameters; and
second decoding code configured to cause the at least one processor to decode the compressed representation of the plurality of blocks based on the updated E2E NIC framework to generate a reconstructed image.
10 . The apparatus of claim 9 , the computer program code further including distortion loss determining code configured to cause the at least one processor to determine a distortion loss of the reconstructed image based on a consumption of the compressed representation of the plurality of blocks and the updated bias parameters, trade-off hyperparameter, and a distortion between block residuals of the compressed representation of the plurality of blocks and a block residual of the decoded compressed representations of the plurality of blocks.
11 . The apparatus of claim 8 , wherein the same pattern is determined based on an RGB variance of the plurality of blocks or a YUV variance of the plurality of blocks.
12 . The apparatus of claim 8 , wherein the updated bias parameters include a learning rate and a number of steps, and the learning rate and the number of steps are selected based on characteristics of the input image.
13 . The apparatus of claim 12 , wherein the characteristics of the input image are one of a RGB variance of the input image and an RD performance of the input image.
14 . The apparatus of claim 8 , wherein when preprocessing the neural network, the neural network is fine-tuned using the plurality of blocks.
15 . A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of an apparatus for content-adaptive online training for end-to-end (E2E) neural image compression (NIC) using a neural network, cause the at least one processor to:
receive, by an E2E NIC framework, an input image comprising a plurality of blocks;
preprocess a neural network of the E2E NIC framework using a preprocessed neural network based on the plurality of blocks;
compute updated bias parameters using a last layer of the preprocessed neural network, wherein the updated bias parameters comprise first updated bias parameters that are associated with a first subset of blocks from the plurality of blocks that share a same pattern, the first subset of blocks having one of a same RGB variance, a same YUV variance, and a same dominant color; and
generate an updated E2E NIC framework, based on the updated bias parameters.
16 . The non-transitory computer-readable medium of claim 15 , wherein the instructions further cause the at least one processor to:
encode the plurality of blocks and the updated bias parameters to generate a compressed representation of the plurality of blocks and a compressed representation of the updated bias parameters;
decode the compressed representation of the updated bias parameters to generate decoded updated bias parameters;
update the E2E NIC framework based on the decoded updated bias parameters; and
decode the compressed representation of the plurality of blocks based on the updated E2E NIC framework to generate a reconstructed image.
17 . The non-transitory computer-readable medium of claim 16 ,
wherein the instructions further cause the at least one processor to determine a distortion loss of the reconstructed image based on a consumption of the compressed representation of the plurality of blocks and the updated bias parameters, trade-off hyperparameter, and a distortion between block residuals of the compressed representation of the plurality of blocks and a block residual of the decoded compressed representations of the plurality of blocks.
18 . The non-transitory computer-readable medium of claim 15 , wherein the same pattern is determined based on an RGB variance of the plurality of blocks or a YUV variance of the plurality of blocks.
19 . The non-transitory computer-readable medium of claim 15 , wherein the updated bias parameters include a learning rate and a number of steps, the learning rate and the number of steps are selected based on characteristics of the input image, and the characteristics of the input image are one of a RGB variance of the input image and an RD performance of the input image.
20 . The non-transitory computer-readable medium of claim 15 , wherein when preprocessing the neural network, the neural network is fine-tuned using the plurality of blocks.