Reordering of database records for improved compression
View Patent ↗According to embodiments of the present invention, apparatus, systems, methods and computer program products for sorting and compressing an unordered set of data records from a structured database are provided. Fields of the unordered set of data records are prioritized based on an impact of those fields to a compression scheme for column-oriented compression. The unordered set of data records are sorted based on the prioritized field(s) with a greatest impact on the performance metric. Data of the sorted data records are compressed according to a compression scheme. In some embodiments, prioritizing the fields may be based on an anticipated level of usage of data within those fields and/or a cost function associated with a performance metric as well as optimization of compression. A performance metric may include a faster computational time, reduced I/O computation, faster scan time, etc.
1. A system comprising:
a memory; and
a processor configured to:
prioritize a plurality of fields of an unordered set of data records from a structured database based on an impact of those fields to a performance metric for accessing data stored in a column-oriented compressed database;
select one or more of the prioritized fields based on priority of the prioritized fields, wherein the priority is based on a frequency of access of the fields;
sort the unordered set of data records based on the selected one or more prioritized fields for compression and optimal performance; and
compress data of the sorted data records according to a compression scheme to increase performance for accessing the data stored in the column-oriented compressed database.
2. The system of claim 1 , wherein the processor is configured to prioritize the fields based on a performance metric as assessed by a cost function, the cost function including minimizing a total amount of time reading and computing on a column as well as the compressed size of data in the column.
3. The system of claim 1 , wherein the processor is configured to sort the unordered set of data records based on a prioritized tuple of sort keys.
4. The system of claim 1 , wherein the unordered set of data records is from a columnar database or a hybrid columnar database.
5. The system of claim 1 , wherein the performance metric is determined based upon historical data.
6. The system of claim 1 , wherein the processor is configured to:
update, based upon monitored performance metrics, relative priorities of the various fields to each other;
decompress a set of ordered compressed records to obtain an unordered set of data records;
sort the unordered set of data records based on the updated field priorities to obtain a second set of sorted data records; and
compress data of the second set of sorted data records according to a compression scheme.
7. A computer program product for providing information to a user, comprising a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code, when executed by a processor, causes the processor to:
prioritize a plurality of fields of an unordered set of data records from a structured database based on an impact of those fields to a performance metric for accessing data stored in a column-oriented compressed database;
select one or more of the prioritized fields based on priority of the prioritized fields, wherein the priority is based on a frequency of access of the fields;
sort the unordered set of data records based on the selected one or more prioritized fields for compression and optimal performance; and
compress data of the sorted data records according to a compression scheme to increase performance for accessing the data stored in the column-oriented compressed database.
8. The computer program product of claim 7 , the computer readable program code further configured to cause the processor to prioritize the fields based on a performance metric as assessed by a cost function, the cost function including minimizing a total amount of time reading and computing on a column as well as the compressed size of data in the column.
9. The computer program product of claim 7 , the computer readable program code further configured to cause the processor to sort the unordered set of data records based on a prioritized tuple of sort keys.
10. The computer program product of claim 7 , wherein the performance metric is determined based upon historical data.
11. The computer program product of claim 7 , the computer readable program code further configured to cause the processor to:
update, based upon monitored performance metrics, relative priorities of the various fields to each other;
decompress a set of ordered compressed records to obtain an unordered set of data records;
sort the unordered set of data records based on the updated field priorities to obtain a second set of sorted data records; and
compress data of the second set of sorted data records according to a compression scheme.
12. The computer program product of claim 7 , wherein the unordered set of data records is from a columnar database or a hybrid columnar database.