Multi-cluster warehouse
A method implementing a fault-tolerant data warehouse using availability zones includes allocating a plurality of processing units to a data warehouse, the processing units located in different availability zones, an availability zone comprising one or more data centers. The method further includes routing a query to a processing unit within the data warehouse, the query having a common session identifier with a query previously provided to the processing unit, the processing unit determined to be caching a data segment associated with a cloud storage resource independent of the plurality of processing units. The method further includes, as a result of monitoring a number of queries running at an input degree of parallelism, determining that the processing capacity of the processing units has reached a threshold; and changing a total number of processing units using the input degree of parallelism and the number of queries.
1. A method of implementing a fault-tolerant data warehouse using availability zones, comprising:
allocating a plurality of processing units to a data warehouse, the plurality of processing units located in different availability zones, an availability zone comprising one or more data centers;
routing, by a processing device, a query to a processing unit within the data warehouse, the query having a common session identifier with a query previously provided to the processing unit, the processing unit determined to be caching a data segment associated with a cloud storage resource, usable by the query, wherein the cloud storage resource is independent of the plurality of processing units;
as a result of monitoring a number of queries running at an input degree of parallelism, determining that a processing capacity of the plurality of processing units has reached a threshold; and
changing a total number of processing units associated with the data warehouse using the input degree of parallelism and the number of queries.
2. The method of claim 1 , wherein determining the processing capacity of the plurality of processing units comprises determining an availability of at least one of:
processor resources for each processing unit; or
memory resources for each processing unit.
3. The method of claim 1 , wherein changing the total number of processing units associated with the data warehouse comprises triggering startup of an additional processing unit in response to determining that the query, in combination with a current workload, does not allow one or more currently allocated processing units to meet a performance metric.
4. The method of claim 3 , wherein the method further comprises determining whether the query can be processed such that the performance metric is met.
5. The method of claim 3 , wherein the performance metric comprises a maximum time period that the query will be queued.
6. The method of claim 1 , wherein changing the total number of processing units comprises adding one or more processing units up to a predetermined maximum number of processing units.
7. The method of claim 1 , wherein changing the total number of processing units comprises removing one or more processing units down to a predetermined minimum number of processing units.
8. The method of claim 1 , wherein changing the total number of processing units comprises decommissioning a processing unit of the plurality of processing units in response to determining that a current workload is serviceable by at least one fewer processing units than the plurality of processing units.
9. The method of claim 1 , wherein routing the query to a processing unit within the data warehouse is based on a workload of each of the plurality of processing units.
10. A system, comprising:
a memory; and
a processing device operatively coupled to the memory, the processing device to:
allocate a plurality of processing units to a data warehouse, the plurality of processing units located in different availability zones, an availability zone comprising one or more data centers;
route a query to a processing unit within the data warehouse, the query having a common session identifier with a query previously provided to the processing unit, the processing unit determined to be caching a data segment associated with a cloud storage resource, usable by the query, wherein the cloud storage resource is independent of the plurality of processing units;
as a result of monitoring a number of queries running at an input degree of parallelism, determine that a processing capacity of the plurality of processing units has reached a threshold; and
change a total number of processing units associated with the data warehouse using the number of queries and the input degree of parallelism.
11. The system of claim 10 , wherein the processing device is further to determine the processing capacity of the processing units by determining an availability of at least one of:
processor resources for each processing unit; or
memory resources for each processing unit.
12. The system of claim 10 , wherein to change the total number of processing units, the processing device is further to trigger startup of a new processing unit in response to determining that the query, in combination with a current workload, does not allow one or more currently allocated processing units to meet a performance metric.
13. The system of claim 12 , wherein the processing device is further to determine whether the query can be processed such that the performance metric is met.
14. The system of claim 12 , wherein the performance metric comprises a maximum time period that the query will be queued.
15. The system of claim 10 , wherein to change the total number of processing units, the processing device is further to add one or more processing units up to a predetermined maximum number of processing units.
16. The system of claim 10 , wherein to change the total number of processing units, the processing device is further to remove one or more processing units down to a predetermined minimum number of processing units.
17. The system of claim 10 , wherein to wherein to change the total number of processing units, the processing device is further to decommission a processing unit of the plurality of processing units in response to determining that a current workload is serviceable by at least one fewer processing units than the plurality of processing units.
18. The system of claim 10 , wherein to route the query to a processing unit within the data warehouse, the processing device is further to route the query to the processing unit based on a workload of each of the plurality of processing units.
19. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processing device, cause the processing device to:
allocate a plurality of processing units to a data warehouse, the plurality of processing units located in different availability zones, an availability zone comprising one or more data centers;
route a query to a processing unit within the data warehouse, the query having a common session identifier with a query previously provided to the processing unit, the processing unit determined to be caching a data segment associated with a cloud storage resource, usable by the query, wherein the cloud storage resource is independent of the plurality of processing units;
as a result of monitoring a number of queries running at an input degree of parallelism, determine that a processing capacity of the plurality of processing units has reached a threshold; and
change a total number of processing units associated with the data warehouse using the number of queries and the input degree of parallelism.
20. The non-transitory computer-readable medium of claim 19 , wherein the instructions are further to cause the processing device to determine the processing capacity of the plurality of processing units by determining an availability of at least one of:
processor resources for each processing unit; or
memory resources for each processing unit.
21. The non-transitory computer-readable medium of claim 19 , wherein to change the total number of processing units, the instructions are further to cause the processing device to trigger startup of a new processing unit in response to determining that the query in combination with a current workload does not allow one or more currently allocated processing units to meet a performance metric.
22. The non-transitory computer-readable medium of claim 21 , wherein the instructions are further to cause the processing device to determine whether the query can be processed such that the performance metric is met.
23. The non-transitory computer-readable medium of claim 21 , wherein the performance metric comprises a maximum time period that the query will be queued.
24. The non-transitory computer-readable medium of claim 19 , wherein to change the total number of processing units, the instructions are further to cause the processing device to add one or more processing units up to a predetermined maximum number of processing units.
25. The non-transitory computer-readable medium of claim 19 , wherein to change the total number of processing units, the instructions are further to cause the processing device to remove one or more processing units down to a predetermined minimum number of processing units.
26. The non-transitory computer-readable medium of claim 19 , wherein to change the total number of processing units, the instructions are further to cause the processing device to decommission a processing unit of the plurality of processing units in response to determining that a current workload is serviceable by at least one fewer processing units than the plurality of processing units.