MULTI-CLUSTER WAREHOUSE
A method implementing a fault-tolerant data warehouse using availability zones includes allocating a plurality of processing units to a data warehouse, the processing units located in different availability zones, an availability zone comprising one or more data centers. The method further includes routing a query to a processing unit within the data warehouse, the query having a common session identifier with a query previously provided to the processing unit, the processing unit determined to be caching a data segment associated with a cloud storage resource independent of the plurality of processing units. The method further includes, as a result of monitoring a number of queries running at an input degree of parallelism, determining that the processing capacity of the processing units has reached a threshold; and changing a total number of processing units using the input degree of parallelism and the number of queries.
1 . A method comprising:
allocating a plurality of processing units to a data warehouse, the plurality of processing units located across different availability zones, wherein each availability zone comprises one or more data centers;
routing a query to a first processing unit of the data warehouse, the query having a common session identifier with a query previously provided to the first processing unit, wherein the first processing unit caches a data segment used by the query; and
modifying, by a processing device, a number of processing units allocated to the data warehouse based on a memory usage of each of the plurality of processing units and a degree of concurrency of each of the plurality of processing units.
2 . The method of claim 1 , wherein the degree of concurrency of each of the plurality of processing units is based on an input degree of parallelism of each query the processing units is running.
3 . The method of claim 1 , wherein modifying the number of processing units allocated to the data warehouse comprises triggering startup of one or more additional processing units in response to determining that the query causes a threshold number of the plurality of processing units to operate with a high memory usage and a high degree of concurrency.
4 . The method of claim 1 , wherein modifying the number of processing units comprises adding one or more processing units up to a predetermined maximum number of processing units in response to determining that the query causes a threshold number of the plurality of processing units to operate with a high memory usage and a high degree of concurrency.
5 . The method of claim 1 , wherein modifying the number of processing units allocated to the data warehouse comprises:
modifying the number of processing units allocated to the data warehouse based further on a maximum time period that the query will be queued.
6 . The method of claim 1 , wherein modifying the number of processing units allocated to the data warehouse comprises removing one or more processing units down to a predetermined minimum number of processing units in response to determining that the query causes a threshold number of the plurality of processing units to operate with a low memory usage and a low degree of concurrency.
7 . The method of claim 1 , wherein modifying the total number of processing units comprises removing a processing unit of the plurality of processing units in response to determining that the processing unit has been inactive for a threshold amount of time.
8 . A system comprising:
a memory; and
a processing device operatively coupled to the memory, the processing device to:
allocate a plurality of processing units to a data warehouse, the plurality of processing units located across different availability zones, wherein each availability zone comprises one or more data centers;
route a query to a first processing unit of the data warehouse, the query having a common session identifier with a query previously provided to the first processing unit, wherein the first processing unit caches a data segment used by the query; and
modify a number of processing units allocated to the data warehouse based on a memory usage of each of the plurality of processing units and a degree of concurrency of each of the plurality of processing units.
9 . The system of claim 8 , wherein the degree of concurrency of each of the plurality of processing units is based on an input degree of parallelism of each query the processing units is running.
10 . The system of claim 8 , wherein to modify the number of processing units allocated to the data warehouse, the processing device is to trigger startup of one or more additional processing units in response to determining that the query causes a threshold number of the plurality of processing units to operate with a high memory usage and a high degree of concurrency.
11 . The system of claim 8 , wherein to modify the number of processing units, the processing device is to add one or more processing units up to a predetermined maximum number of processing units in response to determining that the query causes a threshold number of the plurality of processing units to operate with a high memory usage and a high degree of concurrency.
12 . The system of claim 8 , wherein to modify the number of processing units allocated to the data warehouse, the processing device is to:
modify the number of processing units allocated to the data warehouse based further on a maximum time period that the query will be queued.
13 . The system of claim 8 , wherein to modify the number of processing units allocated to the data warehouse, the processing device is to remove one or more processing units down to a predetermined minimum number of processing units in response to determining that the query causes a threshold number of the plurality of processing units to operate with a low memory usage and a low degree of concurrency.
14 . The system of claim 8 , wherein to modify the total number of processing units allocated to the data warehouse, the processing device is to remove a processing unit of the plurality of processing units in response to determining that the processing unit has been inactive for a threshold amount of time.
15 . A non-transitory computer-readable medium having instructions stored thereon which, when executed by the processing device, cause the processing device to:
allocate a plurality of processing units to a data warehouse, the plurality of processing units located across different availability zones, wherein each availability zone comprises one or more data centers;
route a query to a first processing unit of the data warehouse, the query having a common session identifier with a query previously provided to the first processing unit, wherein the first processing unit caches a data segment used by the query; and
modify, by the processing device, a number of processing units allocated to the data warehouse based on a memory usage of each of the plurality of processing units and a degree of concurrency of each of the plurality of processing units.
16 . The non-transitory computer-readable medium of claim 15 , wherein the degree of concurrency of each of the plurality of processing units is based on an input degree of parallelism of each query the processing units is running.
17 . The non-transitory computer-readable medium of claim 15 , wherein to modify the number of processing units allocated to the data warehouse, the processing device is to trigger startup of one or more additional processing units in response to determining that the query causes a threshold number of the plurality of processing units to operate with a high memory usage and a high degree of concurrency.
18 . The non-transitory computer-readable medium of claim 15 , wherein to modify the number of processing units, the processing device is to add one or more processing units up to a predetermined maximum number of processing units in response to determining that the query causes a threshold number of the plurality of processing units to operate with a high memory usage and a high degree of concurrency.
19 . The non-transitory computer-readable medium of claim 15 , wherein to modify the number of processing units allocated to the data warehouse, the processing device is to:
modify the number of processing units allocated to the data warehouse based further on a maximum time period that the query will be queued.
20 . The non-transitory computer-readable medium of claim 15 , wherein to modify the number of processing units allocated to the data warehouse, the processing device is to remove one or more processing units down to a predetermined minimum number of processing units in response to determining that the query causes a threshold number of the plurality of processing units to operate with a low memory usage and a low degree of concurrency.