Compression and decompression of weight values

1 . A method, performed by a computing device comprising a processing element, of compressing weight data representing a set of weight values in a neural network, the method, performed by the processing element, comprising:

obtaining the weight data representing the set of weight values, the set of weight values including a plurality of weight values associated with the neural network;

processing the weight data to identify a frequently occurring positive weight value within the set of weight values;

modifying the weight data by replacing each occurrence of the frequently occurring positive weight value within the set of weight values with a replacement value representing the frequently occurring positive weight value, wherein a magnitude of the replacement value is smaller than a magnitude of the frequently occurring positive weight value;

further modifying the weight data by incrementing each positive weight value within the set of weight values that has not been replaced by the replacement value, wherein incrementing each positive weight value within the set of weight values that has not been replaced by the replacement value accommodates the replacement value by ensuring that positive weight values that have not been replaced by the replacement value in the set of weight values have a larger magnitude than the replacement value;

generating modified weight data representing at least:

the set of weight values after the further modifying the weight data; and

the replacement value associated with the frequently occurring positive weight value;

performing a lossless data compression on the modified weight data to form compressed weight data by converting the set of weight values to variable length codes, wherein the compressed weight data represents the replacement value using a variable length code that has a shorter bit length than a variable length code corresponding to the frequently occurring positive weight value that the replacement value replaced; and

transferring the compressed weight data from the processing element to a computer-readable storage medium for storage.

2 . The method according to claim 1 , wherein steps of processing the weight data to identify a frequently occurring weight value within the set of weight values, modifying the weight data by replacing each occurrence of the frequently occurring weight value within the set of weight values with a replacement value with a magnitude smaller than a magnitude of the frequently occurring weight value, further modifying the weight data by incrementing a magnitude of each weight value within the set of weight values that has not been replaced by the replacement value and associating the frequently occurring weight value and the replacement value form a sequence of steps that are repeated to generate a plurality of different replacement values and associated frequently occurring weight values further represented by the modified weight data.

3 . The method according to claim 2 , wherein after each iteration of the sequence of steps, the method comprises a step of measuring a reduction in size of the modified weight data, the method comprising performing additional iterations of the sequence of steps until a measured reduction in size of the modified weight data is less than a predetermined threshold.

4 . The method according to claim 2 , wherein, for an initial iteration of the sequence of steps, the replacement value is zero.

5 . The method according to claim 1 , wherein the set of weight values are numerical values that can take positive or negative values and the method comprises:

processing the weight data to identify a frequently occurring negative weight value within the set of weight values;

modifying the weight data by replacing each occurrence of the frequently occurring negative weight value within the set of weight values with a further replacement value, wherein a magnitude of the further replacement value is smaller than a magnitude of the frequently occurring negative weight value; and

further modifying the weight data by decrementing each negative weight value within the updated set of weight values that has not been replaced by the replacement value to accommodate the replacement value such that negative weight values that have not been replaced by the further replacement value in the set of weight values have a larger magnitude than the further replacement value,

the modified weight data further representing the further replacement value in association with the frequently occurring negative weight value.

6 . The method according to claim 2 , wherein the set of weight values form a numerical sequence and the plurality of different replacement values are the lowest absolute values in the numerical sequence.

7 . The method according to claim 1 , wherein the variable length code representing the replacement value is selected to be the shortest available variable length code.

8 . A method, performed by a weight decoder of a processing element, of decompressing compressed weight data representing a compressed version of modified weight data representing at least: a set of weight values associated with a neural network; and one or more replacement values associated with respective one or more frequently occurring weight values, the method comprising:

receiving the compressed weight data at the weight decoder from a computer-readable storage medium, wherein the compressed weight data has been formed by a lossless data compression of the modified weight data by converting the set of weight values to variable length codes;

identifying, based on the compressed weight data, the one or more replacement values and the respective one or more frequently occurring weight values associated with the one or more replacement values;

identifying, based on the compressed weight data, a weight value from the set of weight values and determining whether the weight value is less than or equal to a count of the one or more replacement values; and

at least partly decompressing the compressed weight data to obtain decompressed weight data representing a decompressed version of the set of weight values, comprising, for each weight value:

if the weight value is less than or equal to the count of the one or more replacement values, identifying the weight value as a replacement value and replacing the weight value with a frequently occurring weight value corresponding to the replacement value, wherein the compressed weight data represents the replacement value using a variable length code that has a shorter bit length than a variable length code corresponding to the weight value to replace the replacement value,

if the weight value is greater than the count of the one or more replacement values, subtracting a value equal to the count of replacement values associated with the set of weight values from the weight value.

9 . The method of decompressing the compressed weight data according to claim 8 , wherein the steps of identifying, based on the compressed weight data, the one or more replacement values and the respective one or more frequently occurring weight values, and replacing each instance of the one or more replacement values in the set of weight values with the respective frequently occurring weight value form a sequence of steps, and the sequence of steps is repeated for each of a plurality of replacement values and corresponding frequently occurring weight values.

10 . The method of decompressing the compressed weight data according to claim 9 , further comprising sequentially decoding the compressed weight data by first loading the one or more replacement values and frequently occurring weight values into a storage of the weight decoder and subsequently reading respective ones of the plurality of weight values from the set of weight values, wherein each time an replacement value is read in the set of weight values being processed, the weight decoder reads the frequently occurring weight value associated with the replacement value from the storage and replaces the replacement value with the associated frequently occurring weight value in the set of weight values.

11 . A processing element comprising a weight decoder adapted to decompress compressed weight data representing a compressed version of modified weight data representing at least: a set of weight values associated with a neural network; and one or more replacement values associated with respective one or more frequently occurring weight values, the processing element adapted to:

receive the compressed weight data at the weight decoder from a computer-readable storage medium, wherein the compressed weight data has been formed by a lossless data compression of the modified weight data by converting the set of weight values to variable length codes;

identify, based on the compressed weight data, the one or more replacement values and the respective one or more frequently occurring weight values associated with the one or more replacement values;

identify, based on the compressed weight data, a weight value of the set of weight values in the compressed set of weight values and determine whether the weight value is less than or equal to a count of the one or more replacement values; and

at least partly decompress the compressed weight data to obtain decompressed weight data representing a decompressed version of the set of weight values, comprising, for each weight value:

if the weight value is less than or equal to the count of the one or more replacement values, identify the weight value as an replacement value and replace the weight value with a frequently occurring weight value corresponding to the replacement value, wherein the compressed weight data represents the replacement value using a variable length code that has a shorter bit length than a variable length code corresponding to the weight value to replace the replacement value,

if the weight value is greater than the count of the one or more replacement values, subtract a value equal to the count of replacement values associated with the set of weight values from the weight value.

12 . The method according to claim 6 , wherein the variable length codes representing the set of weight values are Golomb Rice codes.

13 . The method according to claim 1 , wherein the frequently occurring positive weight value within the set of weight values is the most common weight value within the set of weight values.