Compressing weight updates for decoder-side neural networks
A method, apparatus, and computer program product are provided for training a neural network or providing a pre-trained neural network with the weight-updates being compressible using at least a weight-update compression loss function and/or task loss function. The weight-update compression loss function can comprise a weight-update vector defined as a latest weight vector minus an initial weight vector before training. A pre-trained neural network can be compressed by pruning one or more small-valued weights. The training of the neural network can consider the compressibility of the neural network, for instance, using a compression loss function, such as a task loss and/or a weight-update compression loss. The compressed neural network can be applied within a decoding loop of an encoder side or in a post-processing stage, as well as at a decoder side.
1. A method comprising:
training a neural network on a training dataset, wherein the training comprises applying at least a task loss function and a weight-update compression loss function, the weight-update compression loss function comprising a weight-update vector defined as a latest weight vector minus an initial weight vector before training, the weight-update compression loss function comprising a ratio of an L1-norm of the weight-update vector to an L2-norm of the weight-update vector;
pruning weight-update values which are between zero and a predetermined threshold value, wherein pruning comprises setting the weight-update values to zero; and
quantizing the non-pruned weight-update values and/or the pruned weight-update values.
2. The method of claim 1 , further comprising:
entropy encoding the resulting weight-update vector to obtain an entropy-encoded weight-update vector.
3. The method of claim 1 , wherein the weight-update compression loss function is minimized to increase a sparsity and a quantizability of weight-update values.
4. The method of claim 1 , wherein the weight-update compression loss function is minimized to lower an entropy of the weight-update values.
5. An apparatus comprising:
at least one processor; and
at least one memory storing computer program code, the at least one memory and the computer program code being configured to, with the processor, cause the apparatus to:
train a neural network on a training dataset, wherein the training comprises applying at least a task loss function and a weight-update compression loss function, the weight-update compression loss function comprising a weight-update vector defined as a latest weight vector minus an initial weight vector before training, the weight-update compression loss function comprising a ratio of an L1-norm of the weight-update vector to an L2-norm of the weight-update vector;
prune weight-update values which are between zero and a predetermined threshold value, wherein pruning comprises setting the weight-update values to zero; and
quantize the non-pruned weight-update values and/or the pruned weight-update values.
6. The apparatus of claim 5 , wherein the at least one memory and the computer program code are further configured to, with the processor, to cause the apparatus to:
entropy encode the weight-update vector to obtain an entropy-encoded weight-update vector.
7. The apparatus of claim 5 , wherein the weight-update compression loss function is minimized to increase a sparsity and a quantizability of weight-update values.
8. The apparatus of claim 5 , wherein the weight-update compression loss function is minimized to lower an entropy of the weight-update values.
9. A computer program product comprising a non-transitory computer-readable storage medium having stored thereon computer-readable code, which, when executed by at least one processor, causes the at least one processor to:
train a neural network on a training dataset, wherein the training comprises applying at least a task loss function and a weight-update compression loss function, the weight-update compression loss function comprising a weight-update vector defined as a latest weight vector minus an initial weight vector before training, the weight-update compression loss function comprising a ratio of an L1-norm of the weight-update vector to an L2-norm of the weight-update vector;
prune weight-update values which are between zero and a predetermined threshold value, wherein pruning comprises setting the weight-update values to zero; and
quantize the non-pruned weight-update values and/or the pruned weight-update values.
10. The computer program product of claim 9 , wherein the computer-readable code, when executed by the at least one processor, further causes the at least one processor to:
entropy encode the weight-update vector to obtain an entropy-encoded weight-update vector.
11. The computer program product of claim 9 , wherein the weight-update compression loss function is minimized to increase a sparsity and a quantizability of weight-update values.
12. The computer program product of claim 9 , wherein the weight-update compression loss function is minimized to lower an entropy of the weight-update values.
13. A method comprising:
training a neural network on a training dataset, wherein the training comprises applying at least a task loss function and a weight-update compression loss function, the weight-update compression loss function comprising a weight-update vector defined as a latest weight vector minus an initial weight vector before training, the weight-update compression loss function comprising a ratio of an L1-norm of the weight-update vector to an L2-norm of the weight-update vector;
determining weight-update values which are between zero and a predetermined threshold value;
setting the weight-update values which are between zero and the predetermined threshold value to zero; and
quantizing at least one of the non-zero weight-update values or the zero weight-update values.
14. The method of claim 13 , further comprising:
entropy encoding the resulting weight-update vector to obtain an entropy-encoded weight-update vector.
15. The method of claim 13 , wherein the weight-update compression loss function is minimized to increase a sparsity and a quantizability of weight-update values.
16. The method of claim 13 , wherein the weight-update compression loss function is minimized to lower-an entropy of the weight-update values.
17. An apparatus comprising:
at least one processor; and
at least one memory storing computer program code, the at least one memory and the computer program code being configured to, with the processor, cause the apparatus to:
train a neural network on a training dataset, wherein the training comprises applying at least a task loss function and a weight-update compression loss function, the weight-update compression loss function comprising a weight-update vector defined as a latest weight vector minus an initial weight vector before training, the weight-update compression loss function comprising a ratio of an L1-norm of the weight-update vector to an L2-norm of the weight-update vector;
determine weight-update values which are between zero and a predetermined threshold value;
set the weight-update values which are between zero and the predetermined threshold value to zero; and
quantize at least one of the non-zero weight-update values or the zero weight-update values.
18. The apparatus of claim 17 , wherein the at least one memory and the computer program code are further configured to, with the processor, to cause the apparatus to:
entropy encode the resulting weight-update vector to obtain an entropy-encoded-weight-update vector.
19. The apparatus of claim 17 , wherein the weight-update compression loss function is minimized to increase a sparsity and a quantizability of the weight-update values.
20. The apparatus of claim 17 , wherein the weight-update compression loss function is minimized to lower-an entropy of the weight-update values.