Resource provisioning systems and methods
A method and apparatus managing a set of processors for a set of queries is described. In an exemplary embodiment, a device receives a set of queries for a data warehouse, the set of queries including one or more queries to be processed by the data warehouse. The device further provisions a set of processors from a first plurality of processors, where the set of processors to process the set of queries, and a set of storage resources to store data for the set of queries. In addition, the device monitors a utilization of the set of processors as the set of processors processes the set of queries. The device additionally updates a number of the processors in the set of processors provisioned based on the utilization/Furthermore, the device processes the set of queries using the updated set of processors.
1. A system comprising:
a plurality of storage resources;
a processor, coupled to the plurality of storage resources, to:
monitor resource utilization of an execution platform based on a data processing
request, wherein the execution platform comprises a plurality of virtual warehouses that each share a plurality of remote storage devices;
determine whether an additional virtual warehouse is needed based on a current resource need for the execution platform; and
provision an amount of processing resources for the additional virtual warehouse independently of changing an amount of storage in the plurality of remote storage devices used by the execution platform.
2. The system of claim 1 , wherein each virtual warehouse of the execution platform comprises a cache configured to store data retrieved from the plurality of remote storage devices and a processor that is independent of the plurality of remote storage devices.
3. The system of claim 1 , wherein the processor further to monitor response rates for the data processing requests.
4. The system of claim 1 , wherein the processor further to deactivate the additional virtual warehouse in response to determining that the additional virtual warehouse is no longer needed.
5. The system of claim 1 , wherein the processor further to
receive a notice of an upcoming data processing request; and
provision a new virtual warehouse to execute the upcoming data processing request.
6. The system of claim 5 , wherein the processor further to deactivate the new virtual warehouse upon completion of the upcoming data processing request.
7. The system of claim 5 , wherein the processor further to
determine a predetermined time duration for the upcoming data processing request; and
deactivate the new virtual warehouse after the predetermined time duration.
8. The system of claim 1 , wherein the processor further to receive a request to perform a plurality of data processing activities and automatically provisioning a new virtual warehouse to execute the request in response to receiving the request.
9. The system of claim 1 , wherein the processor further to determine a plurality of discrete tasks to execute each of the data processing requests and assign the plurality of discrete tasks for each of the data processing requests to one or more virtual warehouses of the execution platform.
10. The system of claim 9 , wherein the processor further to generate a statement result for each of the data processing requests and communicate the statement result to a corresponding account.
11. A method comprising:
monitoring resource utilization of an execution platform based on a data processing requests, wherein the execution platform comprises a plurality of virtual warehouses that each share a plurality of remote storage devices;
determining whether an additional virtual warehouse is needed based on a current resource need for the execution platform; and
provisioning, with a processor, an amount of processing resources for the additional virtual warehouse independently of changing an amount of storage in the plurality of remote storage devices used by the execution platform.
12. The method of claim 11 , wherein each virtual warehouse of the execution platform comprises a cache configured to store data retrieved from the plurality of remote storage devices and a processor that is independent of the plurality of remote storage devices.
13. The method of claim 11 , further comprising deactivating the additional virtual warehouse in response to determining that the additional virtual warehouse is no longer needed.
14. The method of claim 11 , further comprising:
receiving a notice of an upcoming data processing request; and
provisioning a new virtual warehouse to execute the upcoming data processing request.
15. The method of claim 14 , further comprising deactivating the new virtual warehouse upon completion of the upcoming data processing request.
16. A non-transitory computer-readable medium storing instructions which, when executed by one or more second processors of a computing device, cause the one or more second processors to:
monitor resource utilization of an execution platform based on a data processing requests, wherein the execution platform comprises a plurality of virtual warehouses that each share a plurality of remote storage devices;
determine whether an additional virtual warehouse is needed based on a current resource need for the execution platform; and
provision, with a first processor, an amount of processing resources for the additional virtual warehouse independently of changing an amount of storage in the plurality of remote storage devices used by the execution platform.
17. The non-transitory computer-readable medium of claim 16 , wherein each virtual warehouse of the execution platform comprises a cache configured to store data retrieved from the plurality of remote storage devices and a processor that is independent of the plurality of remote storage devices.
18. The non-transitory computer-readable medium of claim 16 , wherein the instructions further cause the one or more second processors to:
receive a notice of an upcoming data processing request; and
provision a new virtual warehouse to execute the upcoming data processing request.
19. The non-transitory computer-readable medium of claim 16 , wherein the instructions cause the one or more second processors to:
deactivate the new virtual warehouse upon completion of the upcoming data processing request.
20. The non-transitory computer-readable medium of claim 16 , wherein the instructions cause the one or more second processors to:
receive a request to perform a plurality of data processing activities; and
automatically provision a new virtual warehouse to execute the request in response to receiving the request.