Random access data compression
View Patent ↗Methods, program products, and systems implementing random access data compression are disclosed. Data can be stored in a data structure in compressed or non-compressed form. The data structure can include a header block, one or more data blocks, and one or more index blocks. Each data block can include data compressed using different compression technology. The header block can include searchable references to the data blocks, which can be located in the data structure after the header block. The searchable references permit non-sequential access to the data blocks. The data blocks can be organized independent of a file system structure. The header block can additionally include references to the one or more index blocks, which can expand the references in the header block.
1. A method executed by one or more data processing devices, comprising:
receiving data to be compressed;
creating a header block, an index block, and one or more data blocks, each data block including a compressed portion of the received data, the header block including a reference to the index block and a first portion of an index to the one or more data blocks, the index block including a second portion of the index to the one or more data blocks;
storing the header block in a serialized data structure; and
storing the one or more data blocks in the serialized data structure after the header block, the first portion of the index indicating a location of each of the one or more data blocks that is located before the index block in the serialized data structure, and the second portion of the index indicating a location of each of the one or more data blocks that is located after the index block in the serialized data structure, wherein the index includes a triplet of values, the values including:
a first index identifying an uncompressed offset value;
a second index identifying an uncompressed data size; and
a third index identifying an offset of a data block.
2. The method of claim 1 , where the header block includes a configurable block size of each data block.
3. The method of claim 1 , further comprising accessing the serialized data structure to retrieve the data non-sequentially, including:
receiving a request to decompress a section of the data, the request specifying a location of the section in the data before the data were compressed;
identifying a triplet of values, the first index of the triplet of values corresponding to the location specified in the request;
identifying a data block using the third index in the identified triplet of values;
decompressing the identified data block; and
identifying the section of the data from the decompressed data block.
4. The method of claim 1 , where the index block includes additional triplets of values and a reference to an additional index block, the index block and the additional index block forming a tree data structure for storing references to the one or more data blocks.
5. The method of claim 4 , where the received data include a plurality of log messages, each log message including offset information usable for identifying a location of the log message from the one or more data blocks using the triplets of values.
6. The method of claim 4 , where the serialized data are stored on a file system and are accessible through a block accessor configured to translate a reference to a point of data to a point of the serialized data stored on the file system.
7. The method of claim 1 , where each data block includes a compression identifier identifying a compression mechanism used in compressing the data into the data block.
8. The method of claim 1 , where the one or more data blocks include a first data block and a second data block, the first data block including data compressed using a first compression mechanism, the second data block including data compressed using a second compression mechanism that is different from the first compression mechanism.
9. The method of claim 1 , where the index is file system independent.
10. The method of claim 1 , where storing the one or more data blocks in the data structure after the header block includes storing the one or more data blocks sequentially after the header block.
11. A computer program product stored on a storage device, operable to cause one or more processors to perform operations comprising:
receiving data to be compressed;
creating a header block, an index block, and one or more data blocks, each data block including a compressed portion of the received data, the header block including a reference to the index block and a first portion of an index to the one or more data blocks, the index block including a second portion of the index to the one or more data blocks;
storing the header block in a serialized data structure; and
storing the one or more data blocks in the data structure after the header block, the first portion of the index indicating a location of each of the one or more data blocks that is located before the index block in the serialized data structure, and the second portion of the index indicating a location of each of the one or more data blocks that is located after the index block in the serialized data structure, wherein the index includes a triplet of values, the values including:
a first index identifying an uncompressed offset value;
a second index identifying an uncompressed data size; and
a third index identifying an offset of a data block.
12. The computer program product of claim 11 , where:
the received data include a plurality of log messages, each log message including offset information usable for identifying a location of the log message from the one or more data blocks using the triplets of values.
13. The computer program product of claim 11 , where the header block includes a configurable block size of each data block.
14. The computer program product of claim 11 , where the index block includes additional triplets of values and a reference to an additional index block, the index block and the additional index block forming a tree data structure for storing references to the one or more data blocks.
15. The computer program product of claim 11 , the operations comprising accessing the serialized data structure to retrieve the data non-sequentially, including:
receiving a request to decompress a section of the data, the request specifying a location of the section in the data before the data were compressed;
identifying a triplet of values, the first index of the triplet of values corresponding to the location specified in the request;
identifying a data block using the third index in the identified triplet of values;
decompressing the identified data block; and
identifying the section of the data from the decompressed data block.
16. A system comprising:
one or more processors configured to perform operations comprising:
receiving data to be compressed;
creating a header block, an index block, and one or more data blocks, each data block including a compressed portion of the received data, the header block including a reference to the index block and a first portion of an index to the one or more data blocks, the index block including a second portion of the index to the one or more data blocks;
storing the header block in a serialized data structure; and
storing the one or more data blocks in the data structure after the header block, the first portion of the index indicating a location of each of the one or more data blocks that is located before the index block in the serialized data structure, and the second portion of the index indicating a location of each of the one or more data blocks that is located after the index block in the serialized data structure, wherein the index includes a triplet of values, the values including:
a first index identifying an uncompressed offset value;
a second index identifying an uncompressed data size; and
a third index identifying an offset of a data block.
17. The system of claim 16 , where:
the received data includes a plurality of log messages, each log message including offset information usable for identifying a location of the log message from the one or more data blocks using the triplets of values.
18. The system of claim 16 , where the header block includes a configurable block size of each data block.
19. The system of claim 16 , where the index block includes additional triplets of values and a reference to an additional index block, the index block and the additional index block forming a tree data structure for storing references to the one or more data blocks.
20. The system of claim 16 , the operations comprising accessing the serialized data structure to retrieve the data non-sequentially, including:
receiving a request to decompress a section of the data, the request specifying a location of the section in the data before the data were compressed;
identifying a triplet of values, the first index of the triplet of values corresponding to the location specified in the request;
identifying a data block using the third index in the identified triplet of values;
decompressing the identified data block; and
identifying the section of the data from the decompressed data block.