Resource provisioning systems and methods
Example resource provisioning systems and methods are described. In one implementation, an execution platform accesses multiple remote storage devices. The execution platform includes multiple virtual warehouses, each of which includes a cache to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices. A resource manager is coupled to the execution platform and monitors received data processing requests and resource utilization. The resource manager also determines whether additional virtual warehouses are needed based on the data processing requests and the resource utilization. If additional virtual warehouses are needed, the resource manager provisions a new virtual warehouse.
1. A system comprising a resource manager having one or more hardware processors configured to:
monitor received data processing requests from a plurality of computerized query sources to be executed by an execution platform, the data processing requests directed to database data stored on a plurality of shared storage devices collectively storing the database data, wherein the execution platform comprises a plurality of nodes, each node independent of the plurality of computerized query sources and comprising at least one processor and at least one local cache caching at least a portion of the database data;
monitor query response rates corresponding to the database data;
determine that at least one of additional data storage capacity and additional processing capacity are needed based on the data processing requests and the query response rates; and
increase, in response to the determining, an amount of the data cached by the execution platform by adding one or more nodes to the plurality of nodes, wherein each of the one or more nodes added comprise at least one processor and at least one local cache.
2. The system of claim 1 , wherein each of the plurality of nodes of the execution platform can access each of the plurality of shared storage devices.
3. The system of claim 1 , wherein a number of shared storage devices is independent from a number of nodes of the execution platform.
4. The system of claim 1 , wherein the resource manager is further configured to determine, based on the data processing requests, that at least one of the additional data storage capacity and the additional processing capacity are no longer needed.
5. The system of claim 4 , wherein the resource manager is further programmed to decrease, in response to the determining that the at least one of the additional data storage capacity and the additional processing capacity are no longer needed, the amount of the database data cached by the execution platform by removing at least one node from the plurality of nodes.
6. The system of claim 1 , wherein the plurality of nodes are arranged in a plurality of virtual warehouses.
7. The system of claim 6 , wherein the plurality of virtual warehouses are grouped into a plurality of virtual warehouse groups, and wherein each of the plurality of virtual warehouse groups is associated with a different group of users.
8. The system of claim 6 , wherein the plurality of virtual warehouses are grouped into a plurality of virtual warehouse groups, and wherein each of the plurality of virtual warehouse groups is associated with a different entity.
9. A method comprising:
controlling, by one or more processors, an execution platform comprising a plurality of nodes, each node thereof comprising at least one processor and at least one local cache, the execution platform configured to process database queries corresponding to database data stored by a plurality of shard storage devices independent from the execution platform;
monitoring, by the one or more processors, the data processing requests that originate from a plurality of query sources independent from the execution platform, the data processing requests directed to the database data;
monitoring, by the one or more processors, query response rates corresponding to the database data;
determining, by the one or more processors, that at least one of additional data storage capacity and additional processing capacity are needed based on the data processing requests and the query response rates; and
increasing, by the one or more processors, in response to the determining, an amount of the database data cached by the execution platform by adding one or more nodes to the plurality of nodes of the execution platform, wherein each of the one or more nodes added comprise at least one processor and at least one local cache.
10. The method of claim 9 , further comprising determining, by the one or more processors based on the data processing requests, that at least one of the additional data storage capacity and the additional processing capacity are no longer needed.
11. The method of claim 9 , further comprising decreasing, by the one or more processors in response to the determining that at least one of the additional data storage capacity and the additional processing capacity are no longer needed, the amount of the database data cached by the execution platform by removing nodes at least one node from the plurality of nodes.
12. The method of claim 9 , wherein the determining that at least one of the additional data storage capacity and the additional processing capacity are no longer needed comprises:
determining, by the one or more processors, current query processing delays; and
determining, by the one or more processors, that the current query processing delays exceed a threshold.
13. Non-transitory computer readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to:
allocate storage resources in a storage platform, the storage resources comprising a plurality of shared storage devices storing database data;
allocate computing resources in an execution platform comprising a plurality of nodes, each node comprising at least one processor and at least one local cache, wherein the execution platform is independent from the storage resources and processes queries corresponding to database data stored by the plurality of shared storage devices of the storage platform;
monitor data processing requests that originate from a plurality of query sources independent from the execution platform, the data processing requests directed to the database data;
monitor query response rates corresponding to the database data;
increase or decrease a number of nodes of the execution platform based on the data processing requests and the query response rates; and
increase or decrease the storage resources in the storage platform based on the data processing requests and current data allocation in the plurality of shard storage devices;
wherein the number of nodes of the execution platform is independent from the amount of storage resources in the storage platform.
14. The non-transitory computer readable storage media of claim 13 , wherein the instructions cause the one or more processors to determine that at least a portion of data storage capacity or processing capacity are no longer needed by:
determining current query processing delays; and
determining that the current query processing delays exceed a threshold.
15. The non-transitory computer readable storage media of claim 13 , wherein the instructions cause the one or more processors to arrange the plurality of nodes of the execution platform in a plurality of virtual warehouses.
16. The non-transitory computer readable storage media of claim 15 , wherein the instructions cause the one or more processors to group the plurality of virtual warehouses into a plurality of virtual warehouse groups, and wherein each of the plurality of virtual warehouse groups is associated with a different group of users.
17. The non-transitory computer readable storage media of claim 15 , wherein the instructions cause the one or more processors to group the plurality of virtual warehouses into a plurality of virtual warehouse groups, and wherein each of the plurality of virtual warehouse groups is associated with a different entity.