Systems and methods for improved machine-learned compression
A computer-implemented method for compressing computer-readable data having improved efficiency can include obtaining, by a computing system including one or more computing devices, input data associated with the computing system; and encoding, by the computing system, the input data and added noise from a noisy channel to produce encoded data based at least in part on an encoder model, wherein encoding the input data and added noise includes additively combining the added noise and the input data to obtain noisy input data and rounding the noisy input data by a soft rounding function, the soft rounding function having a sharpness, to produce the encoded data, wherein the machine-learned encoder model is trained on training data, wherein the training data is encoded with the added noise from the noisy channel.
1. A computer-implemented method for compressing computer-readable data having improved efficiency, the method comprising:
obtaining, by a computing system comprising one or more computing devices, input data associated with the computing system; and
encoding, by the computing system, the input data and added noise from a noisy channel to produce encoded data based at least in part on a machine-learned encoder model, wherein encoding the input data and added noise comprises additively combining the added noise and the input data to obtain noisy input data and rounding the noisy input data by a soft rounding function, the soft rounding function having a sharpness, to produce the encoded data;
wherein the machine-learned encoder model is trained on training data, wherein the training data is encoded with the added noise from the noisy channel.
2. The computer-implemented method of claim 1 , wherein the sharpness is progressively increased during training of the machine-learned encoder model.
3. The computer-implemented method of claim 1 , wherein the soft rounding function comprises, in response to an input, a sum of a rounded input and a ratio of a hyperbolic tangent of the sharpness multiplied by a quantizing factor to a hyperbolic tangent of half of the sharpness.
4. The computer-implemented method of claim 1 , wherein the encoder model comprises one or more neural networks.
5. The computer-implemented method of claim 1 , wherein the input data comprises image data.
6. The computer-implemented method of claim 1 , wherein the added noise comprises uniform noise.
7. The computer-implemented method of claim 1 , wherein the noisy channel is differentiable.
8. The computer-implemented method of claim 1 , wherein the noisy input data is quantized by adding uniform noise during training of the machine-learned encoder model and during inference by the machine-learned encoder model by universal quantization.
9. The computer-implemented method of claim 1 , wherein the added noise comprises Gaussian noise.
10. The computer-implemented method of claim 1 , wherein the noisy channel comprises a lattice quantizer and the added noise comprises lattice quantization noise.
11. The computer-implemented method of claim 1 , wherein the machine-learned encoder model is trained by:
during a forward pass, computing a loss by (h(y+u)) for a sampled instance u; and
during a backward pass, replacing the derivative
∂
∂
y
i
h
(
y
i
+
u
i
)
with its expected value, f(y i +0.5)−f(y i −0.5).
12. The computer-implemented method of claim 1 , wherein the encoder model is trained using a training loss, wherein the training loss comprises a sum over all values of an expected value of a logarithm of a probability of the sum of the input data and a corresponding uniform random variable and the expected value of a decoder model.
13. A computer-implemented method of decoding encoded data, the method comprising:
obtaining, by a computing system comprising one or more computing devices, encoded data, the encoded data encoded based at least in part on input data and added noise from a noisy channel, wherein the encoded data is encoded by additively combining the added noise and the input data to obtain noisy input data and rounding the noisy input data by a soft rounding function, the soft rounding function having a sharpness, to produce the encoded data; and
applying, by the computing system, a reconstruction defined by an expected value given a smooth universal quantization approximation to reconstruct approximated input data.
14. The computer-implemented method of claim 13 , wherein a gradient of the approximated input data is computed without any loss by a distortion term including an expected value of a derivative of a loss function.
15. The computer-implemented method of claim 13 , the encoded data encoded by a machine-learned encoder model trained on training data, wherein the training data is encoded with the added noise from the noisy channel.
16. A computer-implemented method of encoding input data with improved computational and statistical efficiency, the method comprising:
obtaining, by a computing system comprising one or more computing devices, input data associated with the computing system;
obtaining, by the computing system, data indicative of a target distribution based at least in part on the input data;
determining, by the computing system, a uniform distribution based at least in part on the target distribution;
obtaining, by the computing system, a plurality of samples from the uniform distribution;
identifying, by the computing system, an index corresponding to a selected sample of the plurality of samples from the uniform distribution such that the selected sample is distributed according to the target distribution; and
communicating, by the computing system, encoded sample representation data, the encoded sample representation data comprising the index corresponding to the selected sample and a decoding factor comprising a subtractive combination of the selected sample and a uniform random variable corresponding to the selected sample.
17. The computer-implemented method of claim 16 , wherein the input data comprises parameters of a machine-learned model, and wherein the target distribution comprises a distribution of the parameters of the machine-learned model.
18. The computer-implemented method of claim 16 , wherein a support of the uniform distribution is at least equal to a support of the target distribution.
19. The computer-implemented method of claim 16 , wherein identifying, by the computing system, the index corresponding to the selected sample of the plurality of samples comprises:
iteratively scoring, for at least one sample of the plurality of samples, the sample based at least in part on the target distribution; and
selecting the sample as the selected sample if a score of the sample is less than or equal to a target score, the target score comprising an exponential sum of the indices of the plurality of samples divided by a maximum value of the target distribution.
20. The computer-implemented method of claim 16 , wherein the selected sample is transformed to a multiplication of the index and a cumulative density function of a marginal distribution of the selected sample and a width of the target distribution.
21. A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining input data associated with the computing system; and
encoding the input data and added noise from a noisy channel to produce encoded data based at least in part on a machine-learned encoder model, wherein encoding the input data and added noise comprises additively combining the added noise and the input data to obtain noisy input data and rounding the noisy input data by a soft rounding function, the soft rounding function having a sharpness, to produce the encoded data;
wherein the machine-learned encoder model is trained on training data, wherein the training data is encoded with the added noise from the noisy channel.
22. The computing system of claim 21 , wherein the sharpness is progressively increased during training of the machine-learned encoder model.
23. The computing system of claim 21 , wherein the soft rounding function comprises, in response to an input, a sum of a rounded input and a ratio of a hyperbolic tangent of the sharpness multiplied by a quantizing factor to a hyperbolic tangent of half of the sharpness.
24. The computing system of claim 21 , wherein the input data comprises image data.
25. The computing system of claim 21 , wherein the noisy channel is differentiable.
26. The computing system of claim 21 , wherein the noisy input data is quantized by adding uniform noise during training of the machine-learned encoder model and during inference by the machine-learned encoder model by universal quantization.
27. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising:
obtaining input data associated with the computing system; and
encoding the input data and added noise from a noisy channel to produce encoded data based at least in part on a machine-learned encoder model, wherein encoding the input data and added noise comprises additively combining the added noise and the input data to obtain noisy input data and rounding the noisy input data by a soft rounding function, the soft rounding function having a sharpness, to produce the encoded data;
wherein the machine-learned encoder model is trained on training data, wherein the training data is encoded with the added noise from the noisy channel.
28. The one or more non-transitory computer-readable media of claim 27 , wherein the sharpness is progressively increased during training of the machine-learned encoder model.
29. The one or more non-transitory computer-readable media of claim 28 , wherein the soft rounding function comprises, in response to an input, a sum of a rounded input and a ratio of a hyperbolic tangent of the sharpness multiplied by a quantizing factor to a hyperbolic tangent of half of the sharpness.
30. The one or more non-transitory computer-readable media of claim 28 , wherein the input data comprises image data.
31. The one or more non-transitory computer-readable media of claim 28 , wherein the noisy channel is differentiable.
32. The one or more non-transitory computer-readable media of claim 28 , wherein the noisy input data is quantized by adding uniform noise during training of the machine-learned encoder model and during inference by the machine-learned encoder model by universal quantization.
33. A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining encoded data, the encoded data encoded based at least in part on input data and added noise from a noisy channel, wherein the encoded data is encoded by additively combining the added noise and the input data to obtain noisy input data and rounding the noisy input data by a soft rounding function, the soft rounding function having a sharpness, to produce the encoded data; and
applying a reconstruction defined by an expected value given a smooth universal quantization approximation to reconstruct approximated input data.
34. The computing system of claim 33 , wherein a gradient of the approximated input data is computed without any loss by a distortion term including an expected value of a derivative of a loss function.
35. The computing system of claim 33 , the encoded data encoded by a machine-learned encoder model trained on training data, wherein the training data is encoded with the added noise from the noisy channel.
36. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising:
obtaining encoded data, the encoded data encoded based at least in part on input data and added noise from a noisy channel, wherein the encoded data is encoded by additively combining the added noise and the input data to obtain noisy input data and rounding the noisy input data by a soft rounding function, the soft rounding function having a sharpness, to produce the encoded data; and
applying a reconstruction defined by an expected value given a smooth universal quantization approximation to reconstruct approximated input data.
37. The one or more non-transitory computer-readable media of claim 36 , wherein a gradient of the approximated input data is computed without any loss by a distortion term including an expected value of a derivative of a loss function.
38. The one or more non-transitory computer-readable media of claim 36 , the encoded data encoded by a machine-learned encoder model trained on training data, wherein the training data is encoded with the added noise from the noisy channel.
39. A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining input data associated with the computing system;
obtaining data indicative of a target distribution based at least in part on the input data;
determining a uniform distribution based at least in part on the target distribution;
obtaining a plurality of samples from the uniform distribution;
identifying an index corresponding to a selected sample of the plurality of samples from the uniform distribution such that the selected sample is distributed according to the target distribution; and
communicating encoded sample representation data, the encoded sample representation data comprising the index corresponding to the selected sample and a decoding factor comprising a subtractive combination of the selected sample and a uniform random variable corresponding to the selected sample.
40. The computing system of claim 39 , wherein the input data comprises parameters of a machine-learned model, and wherein the target distribution comprises a distribution of the parameters of the machine-learned model.
41. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations comprising:
obtaining input data associated with the computing system;
obtaining data indicative of a target distribution based at least in part on the input data;
determining a uniform distribution based at least in part on the target distribution;
obtaining a plurality of samples from the uniform distribution;
identifying an index corresponding to a selected sample of the plurality of samples from the uniform distribution such that the selected sample is distributed according to the target distribution; and
communicating encoded sample representation data, the encoded sample representation data comprising the index corresponding to the selected sample and a decoding factor comprising a subtractive combination of the selected sample and a uniform random variable corresponding to the selected sample.
42. The one or more non-transitory computer-readable media of claim 41 , wherein the input data comprises parameters of a machine-learned model, and wherein the target distribution comprises a distribution of the parameters of the machine-learned model.