Data compression method and apparatus
View Patent ↗An improved data compression method and apparatus is disclosed, particularly for compressing large database tables. A data structure is disclosed which is fully compatible with the traditional DBMS demands, including the random access requirement of RDBMS. The data structure is built on a mixed format physical layout comprising of fixed-sized fields and variable-sized fields which are compressed depending on the size and frequency of the fields. An improved compression ratio is achieved by exploiting redundancy in the mixed format physical layout to encode the column-wise redundancy in the data itself and the correlations among columns. The present invention provides a very fast random access decompression and enables not only greater compression ratios, but also permits flexibility of choosing from a number of compression algorithms.
1. A method for improving compression of data, comprising:
arranging the data on a mixed format physical layout having a plurality of fixed-sized fields, a plurality of variable-sized fields and a plurality of offset slots, the fixed-sized fields being of a first size and the offset slots being of a second size;
dividing the data on the mixed format physical layout into the fixed-sized fields and the variable sized fields; and
compressing the data of the variable sized fields and the fixed-sized fields.
2. The method defined in of claim 1 , further comprising:
storing sizes of the fixed-sized fields in a data dictionary;
storing frequency of the data in the fixed-sized fields and the variable-sized fields in the data dictionary; and
storing information common to all records in the fixed-sized fields and the variable sized fields in the data dictionary.
3. The method of claim 1 , wherein at least one of the fixed-sized fields comprises a field value.
4. The method defined in of claim 1 , wherein at least one of the fixed-sized fields comprise of comprises a field offset.
5. The method of claim 1 , wherein at least one of the fixed-sized fields comprises a pointer into a data dictionary.
6. The method of claim 3 , further comprising:
storing a value of the at least one of the fixed-sized fields in an additional variable-sized field;
coding the value of the at least one of the fixed-sized fields as a field offset pointing to the additional variable-sized field.
7. The method of claim 3 , further comprising:
storing frequently occurring long values of the fields in a data dictionary;
coding a value of one of the variable-sized fields as a field offset by pointing to one of the frequently occurring long values of the fields in the data dictionary.
8. The method claim 1 , further comprising:
coding a value of one of the variable-sized fields by encoding a field offset into one of the offset slots.
9. The method of claim 5 , further comprising: storing frequently occurring long values of the fields in a second data dictionary, wherein the second data dictionary is larger than the data dictionary; and
coding a value of one of the variable-sized fields as a field value pointing into the second data dictionary.
10. A method for improving compression of data, comprising:
arranging the data on a mixed format layout having a plurality of fixed-sized fields, a plurality of variable-sized fields and a plurality of offset slots, the fixed-sized fields being of a first size and the offset slots being of a second size, wherein the data comprises a group of correlated fields;
dividing the data on the mixed format physical layout into the fixed-sized fields and the variable-sized fields; and
compressing the data of the variable-sized fields and the fixed-sized fields.
11. The method of claim 10 , further comprising:
storing sizes of the fixed-sized fields in a data dictionary;
storing frequency of the data in the fixed-sized fields and the variable sized fields in the data dictionary;
storing information common to all records in the fixed-sized fields and the variable sized fields in the data dictionary.
12. The method of claim 10 , wherein at least one of the fixed-sized fields comprises a field value.
13. The method defined in claim 10 , wherein at least one of the fixed-sized fields of comprises a field offset.
14. The method defined in claim 10 , wherein at least one of the fixed-sized fields comprises a pointer into a data dictionary.
15. The method of claim 12 , further comprising:
storing frequently occurring values for the group of correlated fields in a data dictionary; and
coding a frequently occurring value for the group by pointing a field offset, belonging to the group, to the data dictionary.
16. The method of claim 12 , further comprising:
coding an infrequently occurring value for the group, by pointing a field offset, belonging to the group, to a field in a record.
17. A method for retrieving compressed data, comprising:
receiving a request for decompressing the compressed data;
receiving the compressed data on a mixed format physical layout responsive to the request, wherein the mixed format physical layout comprises a plurality of fixed-sized fields, a plurality of variable-sized fields and a plurality of offset slots, the fixed-sized fields being of a first size and the offset slots being of a second size;
searching for a value in the fixed-sized fields; retrieving the value in the fixed-sized fields corresponding to the received compressed data.
18. The method of claim 17 , wherein the retrieving step further comprises:
retrieving a dictionary entry if the value in the fixed-sized fields comprises a dictionary pointer;
retrieving a value starting from a field offset if the value of the fixed field fixed-sized fields comprises a field offset; and
retrieving a same field from a record, if the value of the fixed-sized fields comprises a record offset.
19. An apparatus for improving compression of data, comprising:
means for arranging the data on a mixed format physical layout having a plurality of fixed-sized fields, a plurality of variable-sized fields and a plurality of offset slots, the fixed-sized fields being of a first size and the offset slots being of a second size;
means for dividing the data on the mixed format physical layout into the fixed-sized fields and the variable sized fields; and
means for compressing the data of the variable sized fields and the fixed-sized fields.
20. An apparatus for retrieving compressed data, comprising:
means for receiving a request for decompressing the compressed data;
means for receiving the compressed data on a mixed format physical layout responsive to the request, wherein the mixed format physical layout comprises a plurality of fixed-sized fields, a plurality of variable-sized fields and a plurality of offset slots, the fixed-sized fields being of a first size and the offset slots being of a second size;
searching for a value in the fixed fields;
means for retrieving the value in the fixed fields corresponding to the received compressed data.
21. A compressible computer medium, comprising a plurality of instructions to cause a computer to perform the steps of:
arranging data on a mixed format physical layout having a plurality of fixed-sized fields, a plurality of variable-sized fields and a plurality of offset slots, the fixed-sized fields being of a first size and the offset slots being of a second size;
dividing the data on a mixed format physical layout into the fixed-sized fields and the variable sized fields; and
compressing the data of the variable sized fields and the fixed-sized fields.
22. The compressible computer medium according to claim 21 , wherein the instructions further cause the computer to perform the steps of:
storing sizes of the fixed-sized fields in a data dictionary;
storing frequency of the data in the fixed-sized fields and the variable-sized fields in the data dictionary;
storing information common to all records in the fixed-sized fields and the variable sized fields in the data dictionary.
23. The compressible computer medium of claim 21 , wherein at least one of the fixed-sized fields comprises a field value.
24. The compressible computer medium of claim 21 , wherein at least one of the fixed-sized fields comprises a field offset.
25. The compressible computer medium of claim 22 , wherein at least one of the fixed-sized fields comprises a pointer into the data dictionary.
26. The compressible computer medium according to claim 23 , wherein the instructions further cause the computer to perform the steps of:
storing a value of the at least one of the fixed-sized fields in an additional variable-sized field;
coding the value of the at least one of the fixed-sized fields as a field offset pointing to the additional variable-sized field.
27. The compressible computer medium according to claim 22 , wherein the instructions further cause the computer to perform the steps of:
storing frequently occurring long values of the fields in the data dictionary;
coding a value of one of the variable-sized fields as a field offset pointing into the data dictionary.
28. The compressible computer medium according to claim 25 , wherein the instructions further cause the computer to perform the steps of:
coding a value of one of the variable-sized fields by encoding a field offset into a record.
29. The compressible computer medium according to claim 22 , wherein the instructions further cause the computer to perform the steps of:
storing frequently occurring long values of the fields in a second data dictionary, wherein the second data dictionary is larger than the data dictionary;
coding a value of one of the variable-sized fields as field value pointing into the second data dictionary.