Systems and methods for compressing structured data via latent variable estimation
A computer-implemented method for compressing structured data or semi-structured data is provided. The method comprises: (a) estimating one or more latent variables associated with rows or columns of the structured data or semi-structured data; (b) partitioning the structured data or semi-structured data in one or more blocks according to the one or more one or more latent variables; (c) applying a sequential encoding algorithm to each of the blocks; and (d) appending a compressed encoding of the one or more latent variables.
1 . A computer-implemented method for compressing structured data or semi-structured data, comprising:
(a) estimating latent variables associated with rows or columns of the structured data or semi-structured data;
(b) partitioning the structured data or the semi-structured data into one or more blocks according to the latent variables;
(c) applying a sequential encoding algorithm to each of the one or more blocks; and
(d) appending a compressed encoding of the latent variables to the compressed encoding of each of the one or more blocks.
2 . The computer-implemented method of claim 1 , wherein the latent variables comprise row latent variables associated with the rows and column latent variables associated with the columns.
3 . The computer-implemented method of claim 1 , wherein the latent variables are estimated utilizing a spectral clustering algorithm.
4 . The computer-implemented method of claim 1 , wherein the latent variables are estimated using side information.
5 . The computer-implemented method of claim 4 , wherein the side information comprises a column datatype, a column name, or a row name.
6 . The computer-implemented method of claim 1 , further comprising, prior to (b), reordering the rows and columns of the structured data or semi-structured data based at least in part on the latent variables.
7 . The computer-implemented method of claim 1 , further comprising, prior to (c), generating a serialized block vector for each of the one or more blocks.
8 . Computer-implemented method of claim 7 , wherein the sequential encoding algorithm is applied to each serialized block vector.
9 . The computer-implemented method of claim 8 , wherein applying the sequential encoding algorithm comprises applying a first base compressor to each serialized block vector and a second base compressor to a latent variables of each serialized block vector.
10 . The computer-implemented method of claim 9 , wherein the first base compressor and the second base compressor are different.
11 . The computer-implemented method of claim 1 , wherein the structured data or semi-structured data is associated with hyperspectral imaging, image processing, quantum chemistry, or large language models.
12 . The computer-implemented method of claim 1 , wherein the structured data or semi-structured data is associated with tabular data.
13 . The computer-implemented method of claim 1 , wherein the method achieves an optimal compression rate.
14 . The computer-implemented method of claim 1 , wherein the method reduces a compression rate by at least about 5% compared to frequency-based entropy encoders (ANS), Lempel-Ziv encoders, or finite-state encoders.
15 . A non-transitory computer readable medium comprising machine executable code that, upon execution by one or more computer processors, implements a lossless compression algorithm, comprising:
(a) estimating latent variables associated with rows or columns of a structured data or semi-structured data;
(b) partitioning the structured data or semi-structured data into one or more blocks according to the latent variables;
(c) applying a sequential encoding algorithm to each of the one or more blocks; and
(d) appending a compressed encoding of the latent variables to the compressed encoding of each of the one or more blocks.
16 . The non-transitory computer readable medium of claim 15 , wherein the latent variables comprise row latent variables associated with the rows and column latent variables associated with the columns.
17 . The non-transitory computer readable medium of claim 15 , wherein the latent variables are estimated utilizing a spectral clustering algorithm.
18 . The non-transitory computer readable medium of claim 15 , wherein the latent variables are estimated using side information.
19 . The non-transitory computer readable medium of claim 18 , wherein the side information comprises a column datatype, a column name, or a row name.
20 . A computer program product for compressing structured data or semi-structured data, the computer program product comprising at least one non-transitory computer-readable medium having computer-readable program code portions embodied therein, the computer-readable program code portions comprising:
an executable portion configured to estimate variables associated with rows or columns of the structured data or semi-structured data;
an executable portion configured to partition the structured data or the semi-structured data into one or more blocks according to the latent variables;
an executable portion configured to apply a sequential encoding algorithm to each of the one or more blocks; and
an executable portion configured to append a compressed encoding of the latent variables to the compressed encoding of each of the one or more blocks.