Hybrid transactional/analytical processing (HTAP) database with shared smart storage for multiple tenants

1 . A hybrid transactional/analytical processing (HTAP) database formed of a plurality of server computing devices, the HTAP database comprising:

an online transaction processing (OLTP) engine;

an online analytical processing (OLAP) engine provided separately from the OLTP engine; and

a shared storage holding data of multiple tenants, wherein

the shared storage is configured to:

receive a request for a first scan from a first tenant and a request for a second scan from a second tenant;

execute the first scan;

after a predetermined first period of time has elapsed without the first scan finishing, pause the first scan and execute the second scan;

monitor utilization of memory and processing of the shared storage; and

when the utilization of at least one of the memory or the processing is above a threshold, limit a total number of concurrent scans permitted to be performed in the shared storage.

2 . The HTAP database of claim 1 , wherein the shared storage comprises:

an OLTP storage holding data in row format for responding to queries of the OLTP engine, the OLTP storage including:

a log store configured to persist logs; and

a page store configured to store versions of data pages and apply the logs to construct current versions of the data pages; and

an OLAP storage holding data in column format for responding to queries of the OLAP engine, the OLAP storage including:

a delta store configured to record changes made to the OLTP storage; and

a base store configured to implement updates from the delta store in persistent storage; and

a replication framework configured to perform log shipping of the logs from the log store to the delta store.

3 . The HTAP database of claim 2 , wherein the shared storage is configured to, upon detecting that the utilization of the memory of the shared storage is above the threshold, pause the log shipping from the log store to the delta store.

4 . The HTAP database of claim 2 , wherein the shared storage is configured to, upon detecting that the utilization of the memory of the shared storage is above the threshold, perform an urgent flush of data from the logs in the delta store to the base store and clear the logs from the delta store.

5 . The HTAP database of claim 1 , wherein the shared storage is configured to:

pre-allocate an arena in a delta store of the shared storage corresponding to a partition and prevent the arena from being used by other partitions; and

upon determining that the partition has not been updated for a predetermined amount of time, release the arena.

6 . The HTAP database of claim 1 , wherein the shared storage is configured to:

receive a request for a third scan and a request for a fourth scan;

execute the third scan and the fourth scan;

after the third scan has finished before a predetermined second period of time has elapsed since a start of the third scan, return results of the third scan;

after the predetermined second period of time has elapsed since a start of the fourth scan without the fourth scan finishing, return partial results of the fourth scan; and

continue executing the fourth scan and return updated results of the fourth scan.

7 . The HTAP database of claim 1 , wherein the shared storage is configured to store at least one column index.

8 . The HTAP database of claim 1 , wherein the shared storage is configured to:

receive a request for a search for a value in an IN-list predicate of data;

if a data type of the data is fixed length, apply a branchless binary search algorithm to perform the search; and

if the data type of the data is variable length, apply a hash search algorithm to perform the search.

9 . The HTAP database of claim 1 , wherein the shared storage is configured to:

receive a request for a first predicate pushdown and a request for a second predicate pushdown; and

determine an evaluation order of the first and second predicate pushdowns according to set rules.

10 . The HTAP database of claim 1 , wherein the shared storage is configured to:

receive a request to delete one or more rows of a DiskRowSet; and

apply a roaring bitmap as a delete bitmap of the DiskRowSet marking the one or more rows as deleted.

11 . A data processing method for a hybrid transactional/analytical processing (HTAP) database including an online transaction processing (OLTP) engine, an online analytical processing (OLAP) engine provided separately from the OLTP engine, and a shared storage holding data of multiple tenants, the method comprising, at the shared storage:

receiving a request for a first scan from a first tenant and a request for a second scan from a second tenant;

executing the first scan;

after a predetermined first period of time has elapsed without the first scan finishing, pausing the first scan and executing the second scan;

monitoring utilization of memory and processing of the shared storage; and

when the utilization of at least one of the memory or the processing is above a threshold, limiting a total number of concurrent scans permitted to be performed in the shared storage.

12 . The method of claim 11 , further comprising, upon detecting that the utilization of the memory of the shared storage is above the threshold, pausing log shipping from a log store of the shared storage to a delta store of the shared storage.

13 . The method of claim 11 , further comprising, upon detecting that the utilization of the memory of the shared storage is above the threshold, perform an urgent flush of data from logs in a delta store of the shared storage to a base store of the shared storage and clear the logs from the delta store.

14 . The method of claim 11 , further comprising:

pre-allocating an arena in a delta store of the shared storage corresponding to a partition and preventing the arena from being used by other partitions; and

upon determining that the partition has not been updated for a predetermined amount of time, releasing the arena.

15 . The method of claim 11 , further comprising:

receiving a request for a third scan and a request for a fourth scan;

executing the third scan and the fourth scan;

after the third scan has finished before a predetermined second period of time has elapsed since a start of the third scan, returning results of the third scan;

after the predetermined second period of time has elapsed since a start of the fourth scan without the fourth scan finishing, returning partial results of the fourth scan; and

continuing executing the fourth scan and returning updated results of the fourth scan.

16 . The method of claim 11 , further comprising:

receiving a request for a search for a value in an IN-list predicate of data;

if a data type of the data is fixed length, applying a branchless binary search algorithm to perform the search; and

if the data type of the data is variable length, applying a hash search algorithm to perform the search.

17 . The method of claim 11 , further comprising:

receiving a request to delete one or more rows of a DiskRowSet; and

applying a roaring bitmap as a delete bitmap of the DiskRowSet marking the one or more rows as deleted.

18 . A hybrid transactional/analytical processing (HTAP) database formed of a plurality of server computing devices, the HTAP database comprising:

an online transaction processing (OLTP) engine;

an online analytical processing (OLAP) engine provided separately from the OLTP engine; and

a shared storage holding data of multiple tenants, the shared storage comprising:

an OLTP storage holding data in row format for responding to queries of the OLTP engine, the OLTP storage including:

a log store configured to persist logs; and

a page store configured to store versions of data pages and apply the logs to construct current versions of the data pages; and

an OLAP storage holding data in column format for responding to queries of the OLAP engine, the OLAP storage including:

a delta store configured to record changes made to the OLTP storage; and

a base store configured to implement updates from the delta store in persistent storage; and

a replication framework configured to perform log shipping of the logs from the log store to the delta store, wherein

the shared storage is configured to:

receive a request for a first scan from a first tenant and a request for a second scan from a second tenant;

execute the first scan;

after a predetermined first period of time has elapsed without the first scan finishing, pause the first scan and execute the second scan;

upon detecting that memory usage of the shared storage is above a first threshold, pause the log shipping from the log store to the delta store; and

upon detecting that memory usage of the shared storage is above a second threshold, perform an urgent flush of data from the logs in the delta store to the base store and clear the logs from the delta store.

19 . The method of claim 11 , further comprising storing at least one column index in the shared storage.

20 . The method of claim 11 , further comprising:

receiving a request for a first predicate pushdown and a request for a second predicate pushdown; and

determining an evaluation order of the first and second predicate pushdowns according to set rules.