Unified architecture for hybrid database storage using fragments
Data records of a data set can be stored in multiple main part fragments retained in on-disk storage. Each fragment can include a number of data records that is equal to or less than a defined maximum fragment size. Using a compression that is optimized for each fragment, each fragment can be compressed. After reading at least one of the fragments into main system memory from the on-disk storage, an operation can be performed on the fragment or fragments while the in the main system memory.
1. A computer program product comprising a non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
storing, in a plurality of fragments retained in on-disk storage, a plurality of data records that comprise a data set, each fragment of the plurality of fragments comprising a local row index, wherein the local row index comprises a search structure to identify each fragment based on RowID intervals, wherein the local row index is mapped to RowID values in a global RowID column for the data set;
compressing a first fragment of the plurality of fragments, wherein compressing comprises using a compression that is optimized for the first fragment;
reading data records in the first fragment into a main system memory from the on-disk storage;
creating a temporary, in-memory data structure including data records from the first fragment and a committed row from a change associated with the data set in the first fragment;
compressing the temporary, in-memory data structure in the main system memory; and
storing data from the compressed temporary, in-memory data structure to a revised fragment in the on-disk storage.
2. The computer program product as in claim 1 , wherein the RowID values in the global RowID column are unique and comprise logical references to the plurality of data records of the data set.
3. The computer program product as in claim 1 , wherein the local row index of a fragment comprises physical locations of data records in the fragment, wherein a maximum fragment size of the fragment being smaller than an available amount of main system memory.
4. The computer program product as in claim 1 , wherein each of the plurality of fragments comprises at least one row set arranged in a temporal order, and wherein a number of data records in the plurality of fragments is equal to or less than a defined fragment size.
5. The computer program product as in claim 1 , wherein the operations further comprise copying one or more uncommitted rows from the change of the data set to a delta part of a record.
6. The computer program product as in claim 5 , wherein the operations further comprise storing cleanup pointers, the cleanup pointers indicating start and stop locations for the temporary data structure.
7. A system comprising:
at least one processor; and
a machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
storing, in a plurality of fragments retained in on-disk storage, a plurality of data records that comprise a data set, each fragment of the plurality of fragments comprising a local row index, wherein the local row index comprises a search structure to identify each fragment based on RowID intervals, wherein the local row index is mapped to RowID values in a global RowID column for the data set;
compressing a first fragment of the plurality of fragments, wherein compressing comprises using a compression that is optimized for the first fragment;
reading data records in the first fragment into a main system memory from the on-disk storage; and
creating a temporary, in-memory data structure including data records from the first fragment and a committed row from a change associated with the data set in the first fragment;
compressing the temporary, in-memory data structure in the main system memory; and
storing data from the compressed temporary, in-memory data structure to a revised fragment in the on-disk storage.
8. The system as in claim 7 , wherein the RowID values in the global RowID column are unique and comprise logical references to the plurality of data records of the data set.
9. The system as in claim 7 , wherein the local row index of a fragment comprises physical locations of data records in the fragment, a maximum fragment size of the fragment being smaller than an available amount of main system memory.
10. The system as in claim 7 , wherein each of the plurality of fragments comprises at least one row set arranged in a temporal order, and wherein a number of data records in the plurality of fragments is equal to or less than a defined fragment size.
11. The system as in claim 7 , wherein the operations further comprise copying one or more uncommitted rows from the change of the data set to a delta part of a record.
12. The system as in claim 11 , wherein the operations further comprise storing cleanup pointers, the cleanup pointers indicating start and stop locations for the temporary data structure.
13. A computer-implemented method comprising:
storing, in a plurality of fragments retained in on-disk storage, a plurality of data records that comprise a data set, each fragment of the plurality of fragments comprising a local row index, wherein the local row index comprises a search structure to identify each fragment based on RowID intervals, wherein the local row index is mapped to RowID values in a global RowID column for the data set;
compressing a first fragment of the plurality of fragments, wherein compressing comprises using a compression that is optimized for the first fragment;
reading data records in the first fragment into a main system memory from the on-disk storage; and
creating a temporary, in-memory data structure including data records from the first fragment and a committed row from a change associated with the data set in the first fragment;
compressing the temporary, in-memory data structure in the main system memory; and
storing data from the compressed temporary, in-memory data structure to a revised fragment in the on-disk storage.
14. The computer-implemented method as in claim 13 , wherein the RowID values in the global RowID column are unique and logical references to the plurality data records of the data set.
15. The computer-implemented method as in claim 13 , wherein the local row index of a fragment comprises physical locations of data records in the fragment, a maximum fragment size of the fragment being smaller than an available amount of main system memory.
16. The computer-implemented method as in claim 13 , wherein each of the plurality of fragments comprises at least one row set arranged in a temporal order, and wherein a number of data records in the plurality of fragments is equal to or less than a defined fragment size.
17. The computer-implemented method as in claim 13 , further comprising copying one or more uncommitted rows from the change of the data set to a delta part of a record.