Native copy mechanism for in-memory and paged primitive data structures
A database system detects a query targeting a first database object stored in a unified persistence format in a persistence store. In response to detecting the query, the database system determines whether to load the first database object in column-loadable format or page-loadable format into an in-memory store. In response to determining to load the first database object in the column-loadable format, the database system determines if a first bit representation of one or more corresponding primitives in the persistence store is compatible with a second bit representation used for storing the primitives in the in-memory store. The database system performs a native copy operation to copy the primitives from the persistence store to the in-memory store if the bit representations are compatible, where the native copy operation involves copying the primitives on a page-by-page basis without decoding and encoding values of the one or more primitives.
1 . A method performed by a database system, comprising:
detecting a load operation for loading a first database object into an in-memory store in a column-loadable format;
responsive to detecting the load operation:
accessing page-based metadata associated with the first database object in a persistence store, and accessing column-based metadata associated with the in-memory store;
comparing the page-based metadata to the column-based metadata;
determining, based on the comparison of the page-based metadata to the column-based metadata, whether an in-memory representation of the first database object will be compatible with a persistence store representation of the first database object; and
responsive to determining that the in-memory representation of the first database object will be compatible with the persistence store representation of the first database object, performing a native copy operation to copy one or more paged primitives representing the first database object into the in-memory store.
2 . The method of claim 1 , wherein the native copy operation comprises copying the one or more paged primitives on a page-by-page basis without decoding and encoding values of the one or more paged primitives.
3 . The method of claim 1 , wherein determining if the in-memory representation of the first database object will be compatible with the persistence store representation of the first database object is performed on a per-primitive basis.
4 . The method of claim 1 , further comprising determining if a first number of rows per column used for the in-memory representation are equal to a second number of rows per column used for the persistence store representation.
5 . The method of claim 1 , further comprising determining if a first number of bits per row element used for the in-memory representation are equal to a second number of bits per row element used for the persistence store representation.
6 . The method of claim 1 , wherein responsive to determining that the in-memory representation of the first database object will be incompatible with the persistence store representation of the first database object, the method further comprising performing a non-native copy operation to copy the one or more paged primitives representing the first database object into the in-memory store.
7 . The method of claim 1 , further comprising locating a contiguous memory buffer that stores the one or more paged primitives.
8 . The method of claim 1 , wherein the one or more paged primitives comprise a paged bit vector.
9 . The method of claim 1 , wherein the one or more paged primitives comprise a paged index vector.
10 . The method of claim 9 , wherein the paged index vector is a bit-compressed index vector.
11 . A system, comprising:
at least one processor; and
at least one memory including program instructions which when executed by the at least one processor causes operations comprising:
detecting a load operation for loading a first database object into an in-memory store in a column-loadable format;
responsive to detecting the load operation:
accessing page-based metadata associated with the first database object in a persistence store, and accessing column-based metadata associated with the in-memory store;
comparing the page-based metadata to the column-based metadata;
determining, based on the comparison of the page-based metadata to the column-based metadata, whether an in-memory representation of the first database object will be compatible with a persistence store representation of the first database object; and
responsive to determining that the in-memory representation of the first database object will be compatible with the persistence store representation of the first database object, performing a native copy operation to copy one or more paged primitives representing the first database object into the in-memory store.
12 . The system of claim 11 , wherein the native copy operation comprises copying the one or more paged primitives on a page-by-page basis without decoding and encoding values of the one or more paged primitives.
13 . The system of claim 11 , wherein determining if the in-memory representation of the first database object will be compatible with the persistence store representation of the first database object is performed on a per-primitive basis.
14 . The system of claim 11 , wherein the operations further comprise determining if a first number of rows per column used for the in-memory representation are equal to a second number of rows per column used for the persistence store representation.
15 . The system of claim 11 , wherein the operations further comprise determining if a first number of bits per row element used for the in-memory representation are equal to a second number of bits per row element used for the persistence store representation.
16 . The system of claim 11 , wherein responsive to determining that the in-memory representation of the first database object will be incompatible with the persistence store representation of the first database object, the operations further comprise performing a non-native copy operation to copy the one or more paged primitives representing the first database object into the in-memory store.
17 . The system of claim 11 , wherein the operations further comprise locating a contiguous memory buffer that stores the one or more paged primitives.
18 . The system of claim 11 , wherein the one or more paged primitives comprise a paged bit vector.
19 . The system of claim 11 , wherein the one or more paged primitives comprise a paged index vector.
20 . A non-transitory computer readable storage medium storing instructions, which when executed by at least one data processor, cause operations comprising:
detecting a load operation for loading a first database object into an in-memory store in a column-loadable format;
responsive to detecting the load operation:
accessing page-based metadata associated with the first database object in a persistence store, and accessing column-based metadata associated with the in-memory store;
comparing the page-based metadata to the column-based metadata;
determining, based on the comparison of the page-based metadata to the column-based metadata, whether an in-memory representation of the first database object will be compatible with a persistence store representation of the first database object; and
responsive to determining that the in-memory representation of the first database object will be compatible with the persistence store representation of the first database object, performing a native copy operation to copy one or more paged primitives representing the first database object into the in-memory store.