Apparatus, method and computer program product for optimizing parameters of a compressed representation of a neural network
In example embodiments, an apparatus, a method, and a computer program product are provided. An example apparatus include processing circuitry; and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the processing circuitry, cause the apparatus at least to: overfit a neural network on each media item, from a batch of media items, for a number of iterations to obtain an overfitted neural network model for the each media item; evaluate the overfitted neural network model on the each media item to obtain evaluation errors; and update parameters of the neural network to be based on the evaluation errors.
1. An apparatus comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to:
overfit a neural network on each media item, from a batch of media items, for a number of iterations to obtain an individual overfitted neural network model for each media item in the batch of media items, wherein to overfit the neural network, the following are performed:
selecting overfitting parameters;
performing a forward operation on the neural network;
computing loss based on the forward operation;
computing derivatives of a loss based on the selected overfitting parameters; and
updating the selected overfitting parameters based on the computed derivatives;
evaluate individual overfitted neural network models on a corresponding media item to obtain evaluation errors; and
update parameters of the neural network based on the evaluation errors.
2. The apparatus of claim 1 , wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to perform a training iteration on the neural network based on the evaluation errors.
3. The apparatus of claim 2 , wherein the training iteration comprises two stages, when overfitting parameters comprise a latent tensor output by an encoder neural network, or the latent tensor output by a quantizer, and wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to perform the two stages.
4. The apparatus of claim 1 , wherein the overfitting parameters comprise at least one of the following: encoder neural network parameters, output of an encoder neural network, output of a quantizer, or at least one decoder neural network parameter.
5. The apparatus of claim 4 , wherein the encoder neural network parameters comprise parameters of convolution layers, parameters of fully-connected layers, or parameters of a batch-normalization layer.
6. The apparatus of claim 4 , wherein the at least one decoder neural network parameter comprises at least one of one or more bias terms of convolution layers, bias terms of fully-connected layers in a decoder neural network, attention-multipliers in an instance an attention mechanism is used in an architecture of the decoder neural network, or a vector for deriving one or more bias terms by multiplying the vector by a weight matrix at a decoder side.
7. The apparatus of claim 4 , wherein the at least one decoder neural network parameter comprises at least one of one or more of a set of scalar values that act as gates or attention-multipliers in specifically designed blocks used a decoder neural network.
8. The apparatus of claim 4 , wherein to perform the overfitting when the overfitting parameters comprise the encoder neural network parameters, the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to:
input the media item to an encoding pipeline to obtain a latent tensor;
input the latent tensor to a probability neural network and to a decoding pipeline;
obtain a loss comprising a rate loss and a reconstruction loss;
compute a derivative of the loss based on the selected overfitting parameters in the encoder neural network;
update the selected overfitting parameters based on derivatives of the loss; and
repeat preceding for updated overfitting parameters for a predefined number of iterations.
9. The apparatus of claim 8 , wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to:
compute an evaluation loss comprising rate losses and reconstruction losses based on the updated overfitting parameters;
compute gradients of the evaluation loss based on neural network parameters; and
use the computed gradients to update the neural network parameters.
10. The apparatus of claim 1 , wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus at least to select the batch of media items as a subset of training media items.
11. The apparatus of claim 1 , wherein a loss comprises a rate loss and reconstruction loss.
12. An apparatus comprising:
at least one processor; and
at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to:
overfit a neural network on each media item, from a batch of media items, for a number of iterations to obtain an individual overfitted neural network model for each media item in the batch of media items;
evaluate individual overfitted neural network models on a corresponding media item to obtain evaluation errors;
update parameters of the neural network based on the evaluation errors;
perform a training iteration on the neural network based on the evaluation errors;
wherein the training iteration comprises two stages, when overfitting parameters comprise a latent tensor output by an encoder neural network, or the latent tensor output by a quantizer, and wherein the at least one memory stores instructions that, when executed by the at least one processor, cause the apparatus to perform the two stages at least by:
running the encoder neural network or the quantizer to obtain an initial tensor; and
performing a meta-learning by overfitting an initial latent tensor and update the neural network based on performance of latent tensor overfitting.
13. A method comprising:
overfitting a neural network on each media item, from a batch of media items, for a number of iterations to obtain an individual overfitted neural network model for each media item in the batch of media items, wherein to overfit the neural network, the following are performed:
selecting overfitting parameters;
performing a forward operation on the neural network;
computing loss based on the forward operation;
computing derivatives of a loss based on the selected overfitting parameters; and
updating the selected overfitting parameters based on the computed derivatives;
evaluating individual overfitted neural network models on a corresponding media item to obtain evaluation errors; and
updating parameters of the neural network based on the evaluation errors.
14. The method of claim 13 , further comprising performing a training iteration on the neural network based on the evaluation errors.
15. The method of claim 13 , wherein the overfitting parameters comprise at least one of the following: encoder neural network parameters, output of an encoder neural network, output of a quantizer, or at least one decoder neural network parameters.
16. The method of claim 15 , wherein the encoder neural network parameters comprise parameters of convolution layers, parameters of fully-connected layers, or parameters of a batch-normalization layer.
17. The method of claim 15 , wherein the at least one decoder neural network parameters comprise at least one of one or more bias terms of convolution layers, bias terms of fully-connected layers in a decoder neural network, attention-multipliers in an instance an attention mechanism is used in an architecture of the decoder neural network, or a vector for deriving one or more bias terms by multiplying the vector by a weight matrix at a decoder side.