Block compression of tables with repeated values
Methods and apparatus, including computer program products, for block compression of tables with repeated values. In general, value identifiers representing a compressed column of data may be sorted to render repeated values contiguous, and block dictionaries may be generated. A block dictionary may be generated for each block of value identifiers. Each block dictionary may include a list of block identifiers, where each block identifier is associated with a value identifier and there is a block identifier for each unique value in a block. Blocks may have standard sizes and block dictionaries may be reused for multiple blocks.
1. A computer program product, embodied in a non-transitory computer-readable storage medium, the computer program product being operable to cause at least one data processing apparatus to perform operations comprising:
storing, in a delta buffer, change indications corresponding to changes to a column of data stored in a database table, wherein the delta buffer is stored separately from the column of data stored in the database table;
asynchronously integrating the change indications stored in the delta buffer with the column of data stored in the database table by at least
merging the change indications stored in the delta buffer with data stored in the column of data to produce merged data, and
saving the merged data to the column of data, wherein the integrating is performed asynchronously with respect to receiving the change indications;
performing block compression of at least the column of data, wherein the block compression is initiated when the column of data includes a value that is repeated more frequently than other values in the column of data;
receiving a search request for at least a portion of the data stored in the column of data; and
providing, in response to the search request, integrated search results comprising at least a portion of the data stored in the column of data and updated based on the change indications stored in the delta buffer.
2. A computer program product as in claim 1 , wherein data is stored in a plurality of columns, wherein the plurality of columns includes the column of data, and wherein the change indications comprise requests to change one or more fields in at least one row of the column of data.
3. A computer program product as in claim 2 , wherein the operations further comprise:
storing, in a plurality of delta buffers, a plurality of change indications corresponding to the plurality of columns, and wherein the plurality of delta buffers are stored separately from the plurality of columns.
4. A computer program product as in claim 3 , wherein the operations further comprise:
performing, for each of the plurality of columns, dictionary-based compression, the dictionary-based compression generating a column of value identifiers, each of the value identifiers representing a unique value in a respective column of data;
sorting values represented in the plurality of columns, the sorting including ordering the plurality of columns such that the plurality of columns are ordered in order of most to least frequently occurring values;
generating a bit vector representation of each of the plurality of columns representing whether a value in each of the plurality of columns occurs frequently;
removing, from the bit vector to obtain a shortened bit vector, bits corresponding to a most frequently occurring value for each of the plurality of columns; and
storing, as compressed data, the most frequently occurring value and the shortened bit vector for each of the plurality of columns.
5. A computer program product as in claim 4 , wherein the compressed data is decompressed when a search is performed.
6. A computer-implemented method comprising:
storing, in a delta buffer, change indications corresponding to changes to a column of data stored in a database table, wherein the delta buffer is stored separately from the column of data stored in the database table;
asynchronously integrating the change indications stored in the delta buffer with the column of data stored in the database table by at least
merging the change indications stored in the delta buffer with data stored in the column of data to produce merged data, and
saving the merged data to the column of data, wherein the integrating is performed asynchronously with respect to receiving the change indications;
performing block compression of at least the column of data, wherein the block compression is initiated when the column of data includes a value that is repeated more frequently than other values in the column of data;
receiving a search request for at least a portion of the data stored in the column of data; and
providing, in response to the search request, integrated search results comprising at least a portion of the data stored in the column of data and updated based on the change indications stored in the delta buffer.
7. A computer-implemented method as in claim 6 , wherein data is stored in a plurality of columns, wherein the plurality of columns includes the column of data, and wherein the change indications comprise requests to change one or more fields in at least one row of the column of data.
8. A computer-implemented method as in claim 7 , further comprising:
storing, in a plurality of delta buffers, a plurality of change indications corresponding to the plurality of columns, and wherein the plurality of delta buffers are stored separately from the plurality of columns.
9. A system comprising:
at least one programmable processor; and
memory storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
storing, in a delta buffer, change indications corresponding to changes to a column of data stored in a database table, wherein the delta buffer is stored separately from the column of data stored in the database table;
asynchronously integrating the change indications stored in the delta buffer with the column of data stored in the database table by at least
merging the change indications stored in the delta buffer with data stored in the column of data to produce merged data, and
saving the merged data to the column of data, wherein the integrating is performed asynchronously with respect to receiving the change indications;
performing block compression of at least the column of data, wherein the block compression is initiated when the column of data includes a value that is repeated more frequently than other values in the column of data;
receiving a search request for at least a portion of the data stored in the column of data; and
providing, in response to the search request, integrated search results comprising at least a portion of the data stored in the column of data and updated based on the change indications stored in the delta buffer.
10. A system as in claim 9 , wherein data is stored in a plurality of columns, wherein the plurality of columns includes the column of data, and wherein the change indications comprise requests to change one or more fields in at least one row of the column of data.
11. A system as in claim 10 , wherein the operations further comprise:
storing, in a plurality of delta buffers, a plurality of change indications corresponding to the plurality of columns, and wherein the plurality of delta buffers are stored separately from the plurality of columns.