OZIP compression and decompression
A method, apparatus, and system for OZIP, a data compression and decompression codec, is provided. OZIP utilizes a fixed size static dictionary, which may be generated from a random sampling of input data to be compressed. Compression by direct token encoding to the static dictionary streamlines the encoding and avoids expensive conditional branching, facilitating hardware implementation and high parallelism. By bounding token definition sizes and static dictionary sizes to hardware architecture constraints such as word size or processor cache size, hardware implementation can be made fast and cost effective. For example, decompression may be accelerated by using SIMD instruction processor extensions. A highly granular block mapping in optional stored metadata allows compressed data to be accessed quickly at random, bypassing the processing overhead of dynamic dictionaries. Thus, OZIP can support low latency random data access for highly random workloads, such as for OLTP systems.
1. A method comprising:
determining a static dictionary from a portion of input data, the static dictionary comprising a plurality of entries up to a maximum number of dictionary entries, each of the plurality of entries mapping a token to a definition having a length up to a maximum byte size that is bounded by a hardware specification;
tokenizing the input data using the static dictionary to generate a packed sequential plurality of tokens, each of the packed sequential plurality of tokens having a fixed token size that is configured to address the maximum number of dictionary entries;
storing the static dictionary and the packed sequential plurality of tokens;
wherein the method is performed by one or more computing devices.
2. The method of claim 1 , wherein the storing further stores metadata including a block offset mapping that indicates, for each of a plurality of sequential data blocks of a defined uncompressed block size, a token offset within the packed sequential plurality of tokens, wherein the plurality of sequential data blocks corresponds to the input data.
3. The method of claim 1 , wherein said determining comprises:
searching said portion of said input data to build a candidate dictionary having a candidate number of dictionary entries greater than the maximum number of dictionary entries, the candidate dictionary including all 1-gram entries and most frequently occurring N-gram entries, wherein N is an integer value from 2 to the maximum byte size;
pruning the candidate dictionary to form the static dictionary having the maximum number of dictionary entries, wherein the pruning is configured to attempt to minimize a size of the packed sequential plurality of tokens.
4. The method of claim 1 , wherein said static dictionary further includes a frequency count for each of the plurality of entries.
5. The method of claim 1 , wherein said portion of said input data is randomly sampled from said input data.
6. The method of claim 1 , wherein the hardware specification is based on a word size of the one or more computing devices.
7. The method of claim 1 , wherein the maximum number of dictionary entries is configured such that each definition of the plurality of entries in the static dictionary can fit within a processor cache of the one or more computing devices.
8. A method comprising:
receiving a static dictionary comprising a plurality of entries up to a maximum number of dictionary entries, each of the plurality of entries mapping a token to a definition having a length up to a maximum byte size that is bounded by a hardware specification;
receiving a packed sequential plurality of tokens, each of the packed sequential plurality of tokens having a fixed token size that is configured to address the maximum number of dictionary entries;
processing the packed sequential plurality of tokens using the static dictionary to write into an output buffer in response to a request;
wherein the method is performed by one or more computing devices.
9. The method of claim 8 , wherein the request is at a requested offset with a requested size, wherein the processing is for a set of tokens of the packed sequential plurality of tokens, and wherein the method further comprises, prior to the processing:
receiving metadata including a block offset mapping that indicates, for each of a plurality of sequential data blocks of a defined uncompressed block size, a token offset within the packed sequential plurality of tokens, wherein the plurality of sequential data blocks corresponds to unpacked output data;
determining, based on the metadata, the set of tokens from the packed sequential plurality of tokens to answer the request.
10. The method of claim 9 , wherein the block offset mapping indicates the token offset by including a number of tokens contained for each of the plurality of sequential data blocks, wherein the number of tokens exclude any spanned tokens from a prior data block.
11. The method of claim 9 , wherein the block offset mapping further indicates, for each of the plurality of sequential data blocks, a span offset to identify a token that spans from a prior data block into said each of the plurality of sequential data blocks.
12. The method of claim 8 , wherein the hardware specification is based on a word size of the one or more computing devices.
13. The method of claim 8 , wherein the maximum number of dictionary entries is configured such that each definition of the plurality of entries in the static dictionary can fit within a processor cache of the one or more computing devices.
14. The method of claim 8 , wherein said processing uses a single instruction, multiple data (SIMD) instruction of the one or more computing devices to concatenate multiple unpacked tokens of a set of tokens of the packed sequential plurality of tokens at a time.
15. The method of claim 8 , wherein said processing writes, from the definition in the static dictionary mapped to each token in a set of tokens of the packed sequential plurality of tokens to a writing pointer referencing the output buffer, a number of bytes equivalent to the maximum byte size, wherein the writing pointer is advanced by the length of the definition in the static dictionary mapped to each token in the set of tokens.
16. A decompression system comprising one or more computing devices including:
a memory comprising:
a static dictionary comprising a plurality of entries up to a maximum number of dictionary entries, each of the plurality of entries mapping a token to a definition having a length up to a maximum byte size that is bounded by a hardware specification;
a packed sequential plurality of tokens, each of the packed sequential plurality of tokens having a fixed token size that is configured to address the maximum number of dictionary entries;
a processor configured to:
process the packed sequential plurality of tokens using the static dictionary to write into an output buffer in response to a request.
17. The decompression system of claim 16 , wherein the processor is configured to process a set of tokens of the packed sequential plurality of tokens, and wherein the memory further comprises:
metadata including a block offset mapping that indicates, for each of a plurality of sequential data blocks of a defined uncompressed block size, a token offset within the packed sequential plurality of tokens, wherein the plurality of sequential data blocks corresponds to unpacked output data;
and wherein prior to the processing, the processor is further configured to:
receive the request to provide the unpacked output data at a requested offset with a requested size;
determine, based on the metadata, the set of tokens from the packed sequential plurality of tokens to answer the request.
18. The decompression system of claim 16 , wherein the hardware specification is based on a word size of the one or more computing devices.
19. The decompression system of claim 16 , wherein the processor is further configured to process a set of tokens of the packed sequential plurality of tokens by writing, from the definition in the static dictionary mapped to each token in the set of tokens to a writing pointer referencing the output buffer, a number of bytes equivalent to the maximum byte size, wherein the writing pointer is advanced by the length of the definition in the static dictionary mapped to each token in the set of tokens.
20. The decompression system of claim 16 , wherein the processor is further configured to process a set of tokens of the packed sequential plurality of tokens by executing a single instruction, multiple data (SIMD) instruction to concatenate multiple unpacked tokens of the set of tokens at a time.
21. One or more non-transitory computer-readable storage media storing one or more sequences of instructions which, when executed by one or more processors, cause:
determining a static dictionary from a portion of input data, the static dictionary comprising a plurality of entries up to a maximum number of dictionary entries, each of the plurality of entries mapping a token to a definition having a length up to a maximum byte size that is bounded by a hardware specification;
tokenizing the input data using the static dictionary to generate a packed sequential plurality of tokens, each of the packed sequential plurality of tokens having a fixed token size that is configured to address the maximum number of dictionary entries;
storing the static dictionary and the packed sequential plurality of tokens.
22. One or more non-transitory computer-readable storage media storing one or more sequences of instructions which, when executed by one or more processors, cause:
receiving a static dictionary comprising a plurality of entries up to a maximum number of dictionary entries, each of the plurality of entries mapping a token to a definition having a length up to a maximum byte size that is bounded by a hardware specification;
receiving a packed sequential plurality of tokens, each of the packed sequential plurality of tokens having a fixed token size that is configured to address the maximum number of dictionary entries;
processing the packed sequential plurality of tokens using the static dictionary to write into an output buffer in response to a request.
23. One or more non-transitory computer-readable storage media storing one or more sequences of instructions which, when executed by one or more processors, cause:
storing, in memory, a static dictionary comprising a plurality of entries up to a maximum number of dictionary entries, each of the plurality of entries mapping a token to a definition having a length up to a maximum byte size that is bounded by a hardware specification;
storing, in memory, a packed sequential plurality of tokens, each of the packed sequential plurality of tokens having a fixed token size that is configured to address the maximum number of dictionary entries;
processing the packed sequential plurality of tokens using the static dictionary to write into an output buffer in response to a request.