Resource provisioning systems and methods
Example resource provisioning systems and methods are described. In one implementation, an execution platform accesses multiple remote storage devices. The execution platform includes multiple virtual warehouses, each of which includes a cache to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices. A resource manager is coupled to the execution platform and monitors received data processing requests and resource utilization. The resource manager also determines whether additional virtual warehouses are needed based on the data processing requests and the resource utilization. If additional virtual warehouses are needed, the resource manager provisions a new virtual warehouse.
1. A system comprising:
a plurality of storage resources;
a processor, coupled to the plurality of storage resources, to:
monitor data processing requests received from a plurality of accounts directed to database data stored in a plurality of remote storage devices;
monitor resource utilization of an execution platform, wherein the execution platform comprises a plurality of virtual warehouses that each share the plurality of remote storage devices, the execution platform configured to access the plurality of remote storage devices;
determine current and future resource needs for the execution platform to execute the data processing requests;
determine whether an additional virtual warehouse is needed based on the current and future resource needs for the execution platform; and
provision an amount of processing resources for the additional virtual warehouse independently of increasing an amount of storage in the plurality of remote storage devices used by the execution platform.
2. The system of claim 1 , wherein each virtual warehouse of the execution platform comprises a storage means for storing data retrieved from the plurality of remote storage devices and a processor that is independent of the plurality of remote storage devices.
3. The system of claim 1 , wherein the processor further to monitor response rates for the data processing requests.
4. The system of claim 1 , wherein the processor further to deactivate the additional virtual warehouse in response to determining that the additional virtual warehouse is no longer needed.
5. The system of claim 1 , wherein the processor further to receive a notice of an upcoming data processing request; and
provision a new virtual warehouse to execute the upcoming data processing request.
6. The system of claim 5 , wherein the processor further to deactivate the new virtual warehouse upon completion of the upcoming data processing request.
7. The system of claim 5 , wherein the processor further to determine a predetermined time duration for the upcoming data processing request; and
deactivate the new virtual warehouse after the predetermined time duration.
8. The system of claim 1 , wherein the processor further to receive a request to perform a plurality of data processing activities; and means for automatically provisioning a new virtual warehouse to execute the request in response to receiving the request.
9. The system of claim 1 , wherein the processor further to determine a plurality of discrete tasks to execute each of the data processing requests; and
assign the plurality of discrete tasks for each of the data processing requests to one or more virtual warehouses of the execution platform.
10. The system of claim 9 , wherein the processor further to generate a statement result for each of the data processing requests; and means for communicating the statement result to a corresponding account.
11. A method comprising:
monitoring data processing requests received from a plurality of accounts directed to database data stored in a plurality of remote storage devices;
monitoring resource utilization of an execution platform comprising a plurality of virtual warehouses that each share the plurality of remote storage devices, the execution platform configured to access the plurality of remote storage devices;
determining current and future resource needs for the execution platform to execute the data processing requests;
determining whether an additional virtual warehouse is needed based on the current and future resource needs for the execution platform; and
provisioning an amount of processing resources for the additional virtual warehouse independently of increasing an amount of storage in the plurality of remote storage devices used by the execution platform.
12. The method of claim 11 , wherein each virtual warehouse of the execution platform comprises a cache configured to store data retrieved from the plurality of remote storage devices and a processor that is independent of the plurality of remote storage devices.
13. The method of claim 11 , further comprising deactivating the additional virtual warehouse in response to determining that the additional virtual warehouse is no longer needed.
14. The method of claim 11 , further comprising:
receiving a notice of an upcoming data processing request; and
provisioning a new virtual warehouse to execute the upcoming data processing request.
15. The method of claim 14 , further comprising deactivating the new virtual warehouse upon completion of the upcoming data processing request.
16. A non-transitory computer-readable medium storing instructions which, when executed by one or more second processors of a computing device, cause the one or more second processors to:
monitor data processing requests received from a plurality of accounts directed to database data stored in a plurality of remote storage devices;
monitor resource utilization of an execution platform comprising a plurality of virtual warehouses that each share the plurality of remote storage devices, the execution platform configured to access the plurality of remote storage devices;
determine current and future resource needs for the execution platform to execute the data processing requests;
determine whether an additional virtual warehouse is needed based on the current and future resource needs for the execution platform; and
provision an amount of processing resources for the additional virtual warehouse independently of increasing an amount of storage in the plurality of remote storage devices used by the execution platform.
17. The non-transitory computer-readable medium of claim 16 , wherein each virtual warehouse of the execution platform comprises a cache configured to store data retrieved from the plurality of remote storage devices and a processor that is independent of the plurality of remote storage devices.
18. The non-transitory computer-readable medium of claim 16 , wherein the instructions further comprise: receiving a notice of an upcoming data processing request; and
provision a new virtual warehouse to execute the upcoming data processing request.
19. The non-transitory computer-readable medium of claim 16 , wherein the instructions further comprise deactivating the new virtual warehouse upon completion of the upcoming data processing request.
20. The non-transitory computer-readable medium of claim 16 , wherein the instructions further comprise: receiving a request to perform a plurality of data processing activities; and
automatically provision a new virtual warehouse to execute the request in response to receiving the request.