System and method to use dictionaries in LZ4 block format compression
An information handling system for compressing data includes a data storage device and a processor. The data storage device stores a dictionary and an uncompressed data block. The processor prepends the dictionary to the uncompressed data block, determines, from the uncompressed data block, a literal data string and a match data string where the match data string is a matching entry of the dictionary, and compresses the uncompressed data block into a compressed data block that includes the literal data string and an offset pointer that points to the matching entry.
1. An information handling system, comprising:
a data storage device including a first dictionary and a first uncompressed data block; and
a processor configured to:
prepend the first dictionary to the first uncompressed data block;
determine, from the first uncompressed data block, a first literal data string and a first match data string, the first match data string being a first matching entry of the first dictionary; and
compress the first uncompressed data block into a first compressed data block that includes the first literal data string and a first offset pointer, the first offset pointer having a first offset that is greater than an offset of a start of the first literal data string and that points to the first matching entry in the prepended first dictionary.
2. The information handling system of claim 1 , wherein the processor is further configured to:
determine, from the first uncompressed data block, a second literal data string and a second match data string, the second match data string being a matching entry of a sliding window; and
further compress the first uncompressed data block into the first compressed data block that further includes the second literal data string and a second offset pointer, the second offset pointer having a second offset that is less than the offset of the start of the first literal data string and that points into the sliding window.
3. The information handling system of claim 1 , wherein the processor is further configured to:
append a second uncompressed data block to the first uncompressed data block;
determine, from the second uncompressed data block, a second literal data string and a second match data string, the second match data string being a second matching entry of the first dictionary; and
compress the second uncompressed data block into the first compressed data block that further includes the second literal data string and a second offset pointer that points to the second matching entry.
4. The information handling system of claim 1 , wherein first block metadata associated with the first uncompressed data block includes information that identifies the first dictionary.
5. The information handling system of claim 4 , wherein second block metadata associated with the first compressed data block includes the information.
6. The information handling system of claim 1 , wherein the first compressed data block includes information that identifies the first dictionary.
7. The information handling system of claim 6 , wherein the information is included in a first in order sequence of the first compressed data block.
8. The information handling system of claim 1 , wherein the processor is further configured to:
prepend a second dictionary to a second uncompressed data block;
determine, from the second uncompressed data block, a second literal data string and a second match data string, the second match data string being a second matching entry of the second dictionary; and
compress the second uncompressed data block into a second compressed data block that includes the second literal data string and a second offset pointer that points to the second matching entry.
9. The information handling system of claim 1 , wherein, in compressing the first uncompressed data block, the processor utilizes an LZ4 data compression algorithm.
10. A data storage device for a distributed storage array, the data storage device comprising:
a data interface for receiving a first dictionary and a first uncompressed data block; and
a processor configured to instantiate a data compressor, the data compressor to:
prepend the first dictionary to the first uncompressed data block;
determine, from the first uncompressed data block, a first literal data string and a first match data string, the first match data string being a first matching entry of the first dictionary; and
compress the first uncompressed data block into a first compressed data block that includes the first literal data string and a first offset pointer, the first offset pointer having a first offset that is greater than an offset of a start of the first literal data string and that points to the first matching entry in the prepended first dictionary.
11. A method, comprising:
prepending a first dictionary to a first uncompressed data block;
determining, from the first uncompressed data block, a first literal data string and a first match data string, the first match data string being a first matching entry of the first dictionary; and
compressing the first uncompressed data block into a first compressed data block that includes the first literal data string and a first offset pointer, the first offset pointer having a first offset that is greater than an offset of a start of the first literal data string and that points to the first matching entry in the prepended first dictionary.
12. The method of claim 11 , further comprising:
determining, from the first uncompressed data block, a second literal data string and a second match data string, the second match data string being a matching entry of a sliding window; and
further compressing the first uncompressed data block into the first compressed data block that further includes the second literal data string and a second offset pointer, the second offset pointer having a second offset that is less than the offset of the start of the first literal data string and that points into the sliding window.
13. The method of claim 11 , further comprising:
receiving the first dictionary from a device configured to perform the compressing.
14. The method of claim 13 , further comprising:
appending a second uncompressed data block to the first uncompressed data block;
determining, from the second uncompressed data block, a second literal data string and a second match data string, the second match data string being a second matching entry of the first dictionary; and
compressing the second uncompressed data block into the first compressed data block that further includes the second literal data string and a second offset pointer that points to the second matching entry.
15. The method of claim 11 , wherein first block metadata associated with the first uncompressed data block includes information that identifies the first dictionary.
16. The method of claim 15 , wherein second block metadata associated with the first compressed data block includes the information.
17. The method of claim 11 , wherein the first compressed data block includes information that identifies the first dictionary.
18. The method of claim 17 , wherein the information is included in a first in order sequence of the first compressed data block.
19. The method of claim 11 , further comprising:
prepending a second dictionary to a second uncompressed data block;
determining, from the second uncompressed data block, a second literal data string and a second match data string, the second match data string being a second matching entry of the second dictionary; and
compressing the second uncompressed data block into a second compressed data block that includes the second literal data string and a second offset pointer that points to the second matching entry.
20. The method of claim 11 , wherein, in compressing the first uncompressed data block, an LZ4 data compression algorithm compressed the first uncompressed data block.