Advanced database compression
A method, a system, and a computer program product for executing a database compression. A compressed string dictionary having a block size and a front coding bucket size is generated from a dataset. Front coding is applied to one or more buckets of strings in the dictionary having the front coding bucket size to generate one or more front coded buckets of strings. One or more portions of the generated front coded buckets of strings are concatenated to form one or more blocks having the block size. Each block is compressed. A set of compressed blocks is stored. The set of the compressed blocks stores all strings in the dataset.
1. A computer-implemented method, comprising:
generating, from a dataset, a compressed string dictionary having a block size and a front coding bucket size;
applying front coding to one or more buckets of strings in the dictionary having the front coding bucket size to generate one or more front coded buckets of strings;
concatenating one or more portions of the generated front coded buckets of strings to form one or more blocks having the block size, each block including a predetermined number of front coded buckets;
compressing each of the one or more blocks; and
storing a set of one or more compressed blocks, wherein the set of the compressed blocks storing all strings in the dataset, wherein the set of one or more compressed blocks is split into one or subsequences of compressed blocks corresponding to respective front coded buckets of strings.
2. The method according to claim 1 , wherein each block in the set of compressed blocks includes a sequence of bucket headers and compressed buckets.
3. The method according to claim 1 , further comprising inserting a termination character between buckets of strings in the plurality of front coded buckets of strings, wherein the termination character is not compressed.
4. The method according to claim 1 , wherein each string includes a string termination character, wherein each string termination character is compressed.
5. The method according to claim 1 , wherein the length of each string is stored and compressed with the string.
6. The method according to claim 1 , wherein the block size is at least one of the following: a fixed size and a variable size.
7. The method according to claim 1 , wherein the compressing further comprises independently executing compression of each block in the plurality of blocks in parallel.
8. The method according to claim 1 , wherein one or more front-coded buckets are configured to be sampled to form one or more superblocks, wherein each one or more superblocks are configured to be compressed to generate a shared grammar for compression of the one or more blocks.
9. The method according to claim 1 , wherein the compressing includes performing re-pair compression of the one or more blocks.
10. The method according to claim 1 , further comprising
receiving a request to access data in the set of stored compressed blocks;
identifying one or more compressed blocks in the set of stored compressed blocks responsive to the received request; and
at least partially decompressing the identified compressed blocks to generate uncompressed data.
11. The method according to claim 10 , wherein the decompressing includes
decompressing string prefixes and suffixes inside the identified compressed blocks using front coding; and
decompressing string suffixes inside the identified compressed blocks using a re-pair decompression.
12. The method according to claim 11 , wherein the re-pair decompression includes iteratively expanding each symbol in a symbol tree corresponding to the data stored in the identified compressed blocks.
13. A system comprising:
at least one programmable processor; and
a non-transitory machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
generating, from a dataset, a compressed string dictionary having a block size and a front coding bucket size;
applying front coding to one or more buckets of strings in the dictionary having the front coding bucket size to generate one or more front coded buckets of strings;
concatenating one or more portions of the generated front coded buckets of strings to form one or more blocks having the block size, each block including a predetermined number of front coded buckets;
compressing each of the one or more blocks; and
storing a set of one or more compressed blocks, wherein the set of the compressed blocks storing all strings in the dataset, wherein the set of one or more compressed blocks is split into one or subsequences of compressed blocks corresponding to respective front coded buckets of strings.
14. The system according to claim 13 , wherein each block in the set of compressed blocks includes a sequence of bucket headers and compressed buckets.
15. The system according to claim 13 , wherein the operations further comprise inserting a termination character between buckets of strings in the plurality of front coded buckets of strings, wherein the termination character is not compressed.
16. The system according to claim 13 , wherein each string includes a string termination character, wherein each string termination character is compressed.
17. The system according to claim 13 , wherein the length of each string is stored and compressed with the string.
18. The system according to claim 13 , wherein the block size is at least one of the following: a fixed size and a variable size.
19. The system according to claim 13 , wherein the compressing further comprises independently executing compression of each block in the plurality of blocks in parallel.
20. A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
generating, from a dataset, a compressed string dictionary having a block size and a front coding bucket size;
applying front coding to one or more buckets of strings in the dictionary having the front coding bucket size to generate one or more front coded buckets of strings;
concatenating one or more portions of the generated front coded buckets of strings to form one or more blocks having the block size, each block including a predetermined number of front coded buckets;
compressing each of the one or more blocks; and
storing a set of one or more compressed blocks, wherein the set of the compressed blocks storing all strings in the dataset, wherein the set of one or more compressed blocks is split into one or subsequences of compressed blocks corresponding to respective front coded buckets of strings.