Compression of data that exhibits mixed compressibility
Systems and methods for compression of data that exhibits mixed compressibility, such as floating-point data, are provided. As one example, aspects of the present disclosure can be used to compress floating-point data that represents the values of parameters of a machine-learned model. Therefore, aspects of the present disclosure can be used to compress machine-learned models (e.g., for reducing storage requirements associated with the model, reducing the bandwidth expended to transmit the model, etc.).
1. A computing system configured to compress data, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining subject data;
performing a first compression technique to compress a first portion of the data and obtain first compressed data, wherein a second portion of the data remains uncompressed following performance of the first compression technique;
transforming the second portion of the data to obtain transformed data;
quantizing the transformed data to obtain quantized data; and
storing the first compressed data and the quantized data.
2. The computing system of claim 1 , wherein the data comprises floating-point data.
3. The computing system of claim 2 , wherein:
the floating-point data comprises one or more sign bits, a plurality of exponent bits, and a plurality of mantissa bits;
the first portion of the data comprises the one or more sign bits, the plurality of exponent bits, and one or more most significant mantissa bits of the plurality of mantissa bits; and
the second portion comprises one or more least significant mantissa bits of the plurality of mantissa bits.
4. The computing system of claim 2 , wherein the floating-point data encodes parameter values for parameters of a machine-learned model.
5. The computing system of claim 1 , wherein transforming the second portion of the data comprises performing a Kashin Decomposition on the second portion of the data.
6. The computing system of claim 1 , wherein quantizing the transformed data comprises applying a uniform quantization to the transformed data.
7. The computing system of claim 1 , wherein quantizing the transformed data comprises truncating at least a portion of the transformed data.
8. The computing system of claim 1 , further comprising:
reconstructing the transformed data from the quantized data; and
reconstructing the second portion from the transformed data.
9. The computing system of claim 8 , wherein reconstructing the transformed data comprises dithering the quantized data.
10. A computer-implemented method of transmitting compressed data, the method comprising:
receiving, at a first computing device, subject data;
determining, by the first computing device, a first portion and a second portion of the subject data;
transforming, by the first computing device, the second portion to obtain transformed data;
quantizing, by the first computing device, the transformed data to obtain quantized data; and
transmitting the quantized data from the first computing device to a second computing device.
11. The computer-implemented method of claim 10 , wherein the subject data comprises floating-point data.
12. The computer-implemented method of claim 11 , wherein the floating-point data comprises a sign component, an exponent component, and a mantissa component.
13. The computer-implemented method of claim 12 , wherein the second portion comprises at least a portion of the mantissa component.
14. The computer-implemented method of claim 12 , wherein the first portion comprises the sign component, the exponent component, and at least a portion of the mantissa component.
15. The computer-implemented method of claim 10 , wherein transforming the second portion comprises performing a Kashin Decomposition on the second portion.
16. The computer-implemented method of claim 10 , wherein quantizing the transformed data comprises performing a uniform quantization on the transformed data.
17. The computer-implemented method of claim 10 , wherein quantizing the transformed data comprises truncating at least a portion of the transformed data.
18. The computer-implemented method of claim 10 , further comprising:
reconstructing, by the second computing device, the transformed data from the quantized data; and
reconstructing, by the second computing device, the second portion from the transformed data.
19. The method of claim 10 , further comprising:
compressing, by the first computing device, the first portion to obtain compressed data;
transmitting the compressed data from the first computing device to the second computing device; and
decompressing, by the second computing device, the compressed data to obtain the first portion.
20. One or more non-transitory computer-readable media that store instructions that, when executed by one or more computing devices, cause the one or more computing devices to compress a machine-learned model by performing operations comprising:
obtaining model parameter data that comprises a plurality of floating-point numbers respectively for a plurality of parameters of the machine-learned model, wherein each of the plurality of floating-point numbers comprises one or more sign bits, a plurality of exponent bits, and a plurality of mantissa bits; and
for each of the plurality of floating-point numbers:
generating first compressed data from the one or more sign bits, the plurality of exponent bits, and one or more most significant mantissa bits of the plurality of mantissa bits to obtain first compressed data;
performing a Kashin Decomposition on one or more remaining least significant mantissa bits to obtain transformed data, the one or more remaining least significant mantissa bits comprising the plurality of mantissa bits which were excluded from the one or more most significant mantissa bits from which the first compressed data was generated;
performing uniform quantization on the transformed data to obtain quantized data; and
storing the first compressed data and the quantized data.