Resource provisioning systems and methods
A system is described inkling a cache memory and a hardware processor. The hardware processor to determine that at least one of additional data storage capacity and additional processing capacity are needed based on data processing requests of a plurality of computerized query sources to be executed by an execution platform comprising a plurality of nodes, where the data processing requests are directed to database data stored on a plurality of shared storage devices collectively storing the database data. The hardware processor further to increase, in response to the determining, an amount of the data cached by the execution platform by adding one or more nodes to the plurality of nodes.
1. A system comprising:
a memory; and
a processor, operatively coupled with the memory, the processor to:
monitor received data processing requests, each data processing request directed to data stored in a plurality of shared storage devices and to be executed by an execution platform comprising a plurality of execution nodes, wherein the plurality of execution nodes is logically divided into one or more virtual warehouses, and each virtual warehouse establishes a communication link with each of one or more of the plurality of shared storage devices based at least in part on a respective received data processing request;
monitor for each of the one or more virtual warehouses, a current resource utilization, query response rates, and a number of users interacting with the virtual warehouse;
determine whether additional processing resources are required based at least in part on the received data processing requests, the current utilization of the one or more virtual warehouses, query response rates of the one or more virtual warehouses and a number of users interacting with each of the one or more virtual warehouses;
determine whether additional data storage capacity is needed based on the received data processing requests;
allocate, in response to determining that additional data storage capacity is needed, one or more additional shared storage devices to the plurality of shared storage devices; and
allocate, in response to determining that additional processing resources are required, one or more additional execution nodes to the plurality of execution nodes.
2. The system of claim 1 , wherein each of the plurality of execution nodes comprise at least one processor and at least one cache, and wherein the current resource utilization of a virtual warehouse comprises a processor utilization and a cache utilization of one or more of the execution nodes of the virtual warehouse.
3. The system of claim 1 , wherein each of the plurality of execution nodes of the execution platform can access each of the plurality of shared storage devices.
4. The system of claim 1 , wherein the plurality of shared storage devices is separate and independent from the execution platform.
5. The system of claim 1 , wherein the processor is further to determine, based on subsequently received data processing requests, that at least one of the additional shared storage devices are no longer needed.
6. The system of claim 5 , wherein the processor is further to decrease, in response to the determining that at least one of the additional shared storage devices are no longer needed, the number of shared storage devices from the plurality of shared storage devices.
7. The system of claim 1 , wherein the one or more virtual warehouses is grouped into one or more virtual warehouse groups, and wherein each of the one or more virtual warehouse groups is associated with a different group of users.
8. The system of claim 1 wherein the one or more virtual warehouses is grouped into one or more virtual warehouse groups, and wherein each of the one or more virtual warehouse groups is associated with a different entity.
9. A method, comprising:
monitoring received data processing requests, each data processing request directed to data stored in a plurality of shared storage devices and to be executed by an execution platform comprising a plurality of execution nodes, wherein the plurality of execution nodes is logically divided into one or more virtual warehouses, and each virtual warehouse establishes a communication link with each of one or more of the plurality of shared storage devices based at least in part on a respective received data processing request;
monitoring, for each of the one or more virtual warehouses, a current resource utilization, query response rates, and a number of users interacting with the virtual warehouse;
determining whether additional processing resources are required based at least in part on the received data processing requests, the current utilization of the one or more virtual warehouses, query response rates of the one or more virtual warehouses and a number of users interacting with each of the one or more virtual warehouses;
determining, by a processor, whether additional data storage capacity is needed based on the received data processing requests;
allocating, in response to determining that additional data storage capacity is needed, one or more additional shared storage devices to the plurality of shared storage devices; and
allocating, in response to determining that additional processing resources are required, one or more additional execution nodes to the plurality of execution nodes.
10. The method of claim 9 , wherein each of the plurality of execution nodes comprise at least one processor and at least one cache, and wherein the current resource utilization of a virtual warehouse comprises a processor utilization and a cache utilization of one or more of the execution nodes of the virtual warehouse.
11. The method of claim 9 , further comprising determining, based on subsequently received data processing requests, that at least one of the additional shared storage devices are no longer needed.
12. The method of claim 11 , wherein the determining that at least one of the shared storage devices are no longer needed comprises:
determining current query processing delays; and
determining that the current query processing delays exceed a threshold.
13. The method of claim 9 , further comprising decreasing, in response to the determining that at least one of the additional shared storage devices are no longer needed, the number of shared storage devices from the plurality of nodes.
14. The method of claim 9 , wherein each of the plurality of execution nodes of the execution platform can access each of the plurality of shared storage devices.
15. The method of claim 9 , wherein the plurality of shared storage devices is independent from the execution platform.
16. The method of claim 9 , wherein the one or more virtual warehouses is grouped into one or more virtual warehouse groups, and wherein each of the one or more virtual warehouse groups is associated with a different group of users.
17. The method of claim 9 , wherein the one or more virtual warehouses is grouped into one or more virtual warehouse groups, and wherein each of the one or more virtual warehouse groups is associated with a different entity.
18. A non-transitory computer readable storage medium storing instructions that, when executed by a processor, cause the processor to:
monitor received data processing requests, each data processing request directed to data stored in a plurality of shared storage devices and to be executed by an execution platform comprising a plurality of execution nodes, wherein the plurality of execution nodes is logically divided into one or more virtual warehouses, and each virtual warehouse establishes a communication link with each of one or more of the plurality of shared storage devices based at least in part on a respective received data processing request;
monitor for each of the one or more virtual warehouses, a current resource utilization, query response rates, and a number of users interacting with the virtual warehouse;
determine whether additional processing resources are required based at least in part on the received data processing requests, the current utilization of the one or more virtual warehouses, query response rates of the one or more virtual warehouses and a number of users interacting with each of the one or more virtual warehouses;
determine whether additional data storage capacity is needed based on data the received data processing requests; and
allocate, in response to determining that additional data storage capacity is needed, one or more additional shared storage devices to the plurality of shared storage devices; and
allocate, in response to determining that additional processing resources are required, one or more additional execution nodes to the plurality of execution nodes.
19. The non-transitory computer readable storage medium of claim 18 , wherein the processor is further to determine, based on subsequently received data processing requests, that at least one of the additional shared storage devices are no longer needed.
20. The non-transitory computer readable storage medium of claim 19 , wherein to determine that at least one of the additional shared storage devices are no longer needed, the processor is to:
determine current query processing delays; and
determine that the current query processing delays exceed a threshold.
21. The non-transitory computer readable storage medium of claim 18 , wherein the processor is further to group the one or more virtual warehouses into one or more virtual warehouse groups, and wherein each of the one or more virtual warehouse groups is associated with a different group of users.
22. The non-transitory computer readable storage medium of claim 18 , wherein the processor is further to group the one or more virtual warehouses into one or more virtual warehouse groups, and wherein each of the one or more virtual warehouse groups is associated with a different entity.