Anisotropic compression as applied to columnar storage formats
Herein are spatially scalable techniques for anisotropic compression of shared entropy between alternate representations of same data. In an embodiment, a computer compresses an uncompressed independent column into a compressed independent column. Based on the compressed independent column, an uncompressed dependent column is compressed into a compressed dependent column. The compressed independent column and the compressed dependent column are stored in a same file. In an embodiment, a computer stores, in metadata, an encrypted private key for decrypting an encrypted column. The encrypted column and the metadata are stored in a file. A request to read the encrypted column is received. Based on a public key and the file, the encrypted private key is decrypted into a decrypted private key. The public key is contained in the request and/or the file. The request is executed by decrypting, based on the decrypted private key and the file, the encrypted column.
1. A method comprising:
compressing an uncompressed independent column into a compressed independent column, wherein the uncompressed independent column and an uncompressed dependent column have shared entropy;
compressing, based on the compressed independent column and said shared entropy, the uncompressed dependent column into a compressed dependent column; and
storing in a file: the compressed independent column, and the compressed dependent column.
2. The method of claim 1 wherein:
said compressing the uncompressed independent column comprises generating primary metadata that describes the compressed independent column;
said compressing the uncompressed dependent column comprises generating secondary metadata that describes the compressed dependent column;
the method further comprises storing, in same said file, between the compressed independent column and the primary metadata: the compressed dependent column, and the secondary metadata.
3. The method of claim 2 wherein the secondary metadata contains provenance metadata that is digitally signed.
4. The method of claim 1 further comprising:
parsing text values into parsed values;
storing the parsed values into the uncompressed independent column;
storing the text values into the uncompressed dependent column.
5. The method of claim 1 wherein the uncompressed dependent column comprises a redaction or masking of the uncompressed independent column.
6. The method of claim 1 wherein:
the uncompressed independent column contains a plurality of values;
a plurality of row-major records contain said plurality of values;
the uncompressed dependent column contains the plurality of row-major records.
7. The method of claim 1 wherein
said compressing the uncompressed independent column and said compressing the uncompressed dependent column concurrently occur.
8. The method of claim 1 further comprising:
compressing, based on the compressed independent column, a second uncompressed dependent column into a second compressed dependent column;
storing the second compressed dependent column in same said file.
9. The method of claim 1 wherein the file is write once.
10. The method of claim 1 wherein:
the uncompressed independent column conforms to a data schema;
the file contains the data schema;
the uncompressed dependent column is schema-less.
11. A method comprising:
tokenizing a text document into separator values and text values;
parsing text values into parsed values;
storing the parsed values into an uncompressed independent column;
storing the separator values into an uncompressed dependent column;
compressing the uncompressed independent column into a compressed independent column;
compressing, based on the compressed independent column, the uncompressed dependent column into a compressed dependent column;
storing in a file: the compressed independent column, and the compressed dependent column, wherein said file does not contain the text document; and
regenerating the text document by decompressing: the compressed independent column and the compressed dependent column.
12. A method comprising:
compressing an uncompressed independent column into a compressed independent column;
compressing, based on the compressed independent column, an uncompressed dependent column into a compressed dependent column, wherein:
the uncompressed independent column consumes less space than the uncompressed dependent column, and
the compressed dependent column consumes less space than the compressed independent column; and
storing in a file: the compressed independent column, and the compressed dependent column.
13. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause:
compressing an uncompressed independent column into a compressed independent column, wherein the uncompressed independent column and an uncompressed dependent column have shared entropy;
compressing, based on the compressed independent column and said shared entropy, the uncompressed dependent column into a compressed dependent column; and
storing in a file: the compressed independent column, and the compressed dependent column.
14. The one or more non-transitory computer-readable media of claim 13 wherein the instructions further cause:
parsing text values into parsed values;
storing the parsed values into the uncompressed independent column;
storing the text values into the uncompressed dependent column.
15. The one or more non-transitory computer-readable media of claim 13 wherein
said compressing the uncompressed independent column and said compressing the uncompressed dependent column concurrently occur.
16. The one or more non-transitory computer-readable media of claim 13 wherein:
the uncompressed independent column conforms to a data schema;
the file contains the data schema;
the uncompressed dependent column is schema-less.
17. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause:
tokenizing a text document into separator values and text values;
parsing text values into parsed values;
storing the parsed values into an uncompressed independent column;
storing the separator values into an uncompressed dependent column;
compressing the uncompressed independent column into a compressed independent column;
compressing, based on the compressed independent column, the uncompressed dependent column into a compressed dependent column;
storing in a file: the compressed independent column, and the compressed dependent column, wherein said file does not contain the text document; and
regenerating the text document by decompressing: the compressed independent column and the compressed dependent column.
18. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause:
compressing an uncompressed independent column into a compressed independent column;
compressing, based on the compressed independent column, an uncompressed dependent column into a compressed dependent column, wherein:
the uncompressed independent column consumes less space than the uncompressed dependent column, and
the compressed dependent column consumes less space than the compressed independent column; and
storing in a file: the compressed independent column, and the compressed dependent column.