Block compression of tables with repeated values
Methods and apparatus, including computer program products, for block compression of tables with repeated values. In general, value identifiers representing a compressed column of data may be sorted to render repeated values contiguous, and block dictionaries may be generated. A block dictionary may be generated for each block of value identifiers. Each block dictionary may include a list of block identifiers, where each block identifier is associated with a value identifier and there is a block identifier for each unique value in a block. Blocks may have standard sizes and block dictionaries may be reused for multiple blocks.
1. A computer program product, tangibly embodied in a computer-readable storage medium, the computer program product being operable to cause data processing apparatus to perform operations comprising:
compressing a column of data out of one or more columns of data in accordance with dictionary-based compression, the compressing comprising generating a column of value identifiers, each of the value identifiers representing a unique value in the column of data;
sorting values represented in the one or more columns of data, the sorting including ordering the one or more columns of data such that the one or more columns of data are ordered in order of frequency of most frequently occurring value in different columns of the one or more columns of data;
generating a bit vector representation of each column representing whether a value in rows of each column occurs frequently;
generating a number representing frequency of the most frequently occurring value for each column;
removing bits corresponding to the most frequently occurring value from the bit vector to obtain a shortened bit vector, the shortened bit vector obtained for each column; and
storing the generated number and the shortened bit vector for each column, the stored number and the shortened bit vector for each column representing the compressed data, the compressed data being decompressed when a search is performed, the decompressed data being compressed again after the search is performed;
wherein changes to the column of the data are stored in a delta buffer separate from the column of the data and the changes are integrated asynchronously.
2. The product of claim 1 , wherein the compressing is initiated when a column of data comprises one value that has a repetition frequency significantly higher than a repetition frequency of other values in the column.
3. The product of claim 1 , wherein the value identifiers are values representing structured business data having data dependencies across a same row of a table.
4. The product of claim 3 , wherein the business data comprises business objects modeled as sets of joined tables.
5. The product of claim 1 , wherein the operations of the product are performed in parallel on a plurality of hardware servers.
6. The product of claim 1 , wherein the search comprises searching for data in the one or more columns of data.
7. A computer-implemented method comprising:
compressing a column of data out of one or more columns of data in accordance with dictionary-based compression, the compressing comprising generating a column of value identifiers, each of the value identifiers representing a unique value in the column of data;
sorting values represented in the one or more columns of data, the sorting including ordering the one or more columns of data such that the one or more columns of data are ordered in order of frequency of most frequently occurring value in different columns of the one or more columns of data;
generating a bit vector representation of each column representing whether a value in each row of the each column occurs frequently;
generating a number representing frequency of the most frequently occurring value for each column;
removing bits corresponding to the most frequently occurring value from the bit vector to obtain a shortened bit vector for each column; and
storing the generated number and the shortened bit vector for each column, the stored number and the shortened bit vector for each column representing the compressed dataU;
wherein changes to the column of the data are stored in a delta buffer separate from the column of the data and the changes are integrated asynchronously.
8. The method of claim 7 further comprising:
decompressing the compressed data when a search is performed, and
recompressing the decompressed data after the search is performed.
9. The method of claim 7 , wherein each column of the one or more columns of data comprises one value that has a repetition frequency significantly higher than a repetition frequency of other values in the each column.
10. The method of claim 7 , wherein the value identifiers are values representing structured business data having data dependencies across a same row of a table, the business data comprising business objects modeled as sets of joined tables.
11. The method of claim 7 , wherein the method is performed in parallel on a plurality of hardware servers.
12. The method of claim 8 , wherein the search comprises a search for data in the one or more columns of data.
13. A system comprising:
at least one programmable processor; and
a machine-readable medium storing instructions that, when executed by the at least one processor, cause the at least one programmable processor to perform operations comprising:
compressing a column of data out of one or more columns of data in accordance with dictionary-based compression, the compressing comprising generating a column of value identifiers, each of the value identifiers representing a unique value in the column of data;
sorting values represented in the one or more columns of data, the sorting including ordering the one or more columns of data such that the one or more columns of data are ordered in order of frequency of most frequently occurring value in different columns of the one or more columns of data;
generating a bit vector representation of each column representing whether a value in each row of the each column occurs frequently;
generating a number representing frequency of the most frequently occurring value for each column of the one or more columns of data; and
removing bits corresponding to the most frequently occurring value from the bit vector to obtain a shortened bit vector for each column, the stored number and the shortened bit vector for each column representing the compressed data
wherein changes to the column of the data are stored in a delta buffer separate from the column of the data and the changes are integrated asynchronously.
14. The system of claim 13 , wherein the processor further performs operations comprising:
storing the generated number and the shortened bit vector for each column.
15. The system of claim 14 , wherein the processor further performs operations comprising:
decompressing the compressed data when a search is performed, and
recompressing the decompressed data after the search is performed.
16. The system of claim 15 , wherein each column of the one or more columns of data comprises one value that has a repetition frequency significantly higher than repetition frequency of other values in the each column.
17. The system of claim 16 , wherein the value identifiers are values representing structured business data having data dependencies across a same row of a table, wherein the business data comprises business objects modeled as sets of joined tables.
18. The system of claim 15 , wherein the search comprises a search for data in the one or more columns of data.