Resource provisioning systems and methods
Example resource provisioning systems and methods are described. In one implementation, an execution platform accesses multiple remote storage devices. The execution platform includes multiple virtual warehouses, each of which includes a cache to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices. A resource manager is coupled to the execution platform and monitors received data processing requests and resource utilization. The resource manager also determines whether additional virtual warehouses are needed based on the data processing requests and the resource utilization. If additional virtual warehouses are needed, the resource manager provisions a new virtual warehouse.
1. A system comprising:
an execution platform configured to access a plurality of remote storage devices, the plurality of remote storage devices collectively storing database data, the execution platform including a plurality of virtual warehouses that each share the plurality of remote storage devices, retrieve data from the plurality of remote storage devices, and execute one or more queries received from client devices against the database data, each virtual warehouse including a cache configured to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices; and
a resource manager coupled to the execution platform and configured to:
determine predicted data processing requests based on previous patterns of previously received data processing requests from a particular client at a particular time;
monitor data processing requests including received data processing requests and the predicted data processing requests;
monitor resource utilization in the plurality of virtual warehouses;
determine whether additional virtual warehouses are needed based on the data processing requests and the resource utilization in the plurality of virtual warehouses; and
responsive to determining that an additional virtual warehouse is needed, provision a new virtual warehouse without increasing an amount of storage in the plurality of remote storage devices.
2. The system of claim 1 , wherein the resource manager is further configured to monitor query response rates.
3. The system of claim 1 , wherein the resource manager is further configured to monitor current resource utilization associated with the plurality of virtual warehouses, and wherein determining whether additional virtual warehouses are needed is further based on the current resource utilization.
4. The system of claim 1 , wherein the resource manager is further configured to monitor at least one of a number of queries running concurrently, a percentage of maximum load, and whether any query processing is delayed.
5. The system of claim 1 , wherein the resource manager is further configured to deactivate the new virtual warehouse responsive to determining that the new virtual warehouse is no longer needed.
6. The system of claim 1 , wherein the resource manager is further configured to:
receive a notice of an upcoming data processing project; and
responsive to receiving the notice of the upcoming data processing project, create a new virtual warehouse associated with the upcoming data processing project.
7. The system of claim 6 , wherein the resource manager is further configured to deactivate the new virtual warehouse upon completion of the data processing project.
8. The system of claim 6 , wherein the upcoming data processing project has a predetermined time duration, and wherein the new virtual warehouse is deactivated after the predetermined time duration.
9. The system of claim 1 , wherein the resource manager is further configured to:
receive a request to perform a plurality of data processing activities; and
responsive to receiving the request to perform the plurality of data processing activities, automatically create a new virtual warehouse associated with the received request.
10. A method comprising:
determining predicted data processing requests based on previous patterns of previously received data processing requests from a particular client at a particular time;
monitoring data processing requests including received data processing requests and the predicted data processing requests, wherein the data processing requests are associated with data stored in a plurality of remote storage devices collectively storing database data;
monitoring resource utilization in a plurality of virtual warehouses, wherein each virtual warehouse includes a cache configured to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices, wherein the plurality of virtual warehouses each share the plurality of remote storage devices, retrieve data from the plurality of remote storage devices, and execute queries against the database data;
determining whether additional virtual warehouses are needed based on the data processing requests and the resource utilization in the plurality of virtual warehouses; and
creating a new virtual warehouse responsive to determining that an additional virtual warehouse is needed without increasing an amount of storage in the plurality of remote storage devices.
11. The method of claim 10 , further comprising monitoring query response rates, wherein determining whether additional virtual warehouses are needed is further based on the query response rates.
12. The method of claim 10 , further comprising monitoring a number of users associated with each of the plurality of virtual warehouses, wherein determining whether additional virtual warehouses are needed is further based on the number of users associated with each of the plurality of virtual warehouses.
13. The method of claim 10 , further comprising:
determining whether the new virtual warehouse is no longer needed; and
deactivating the new virtual warehouse responsive to determining that the new virtual warehouse is no longer needed.
14. The method of claim 10 , further comprising monitoring a number of queries running concurrently, wherein determining whether additional virtual warehouses are needed is further based on the number of queries running concurrently.
15. The method of claim 10 , further comprising determining whether execution of any queries are currently delayed, wherein determining whether additional virtual warehouses are needed is further based on whether any queries are currently delayed.
16. The method of claim 10 , further comprising:
receiving a notice of an upcoming data processing project;
creating a new virtual warehouse associated with the upcoming data processing project; and
deactivating the new virtual warehouse upon completion of the data processing project.
17. The method of claim 10 , further comprising:
receiving a notice of an upcoming data processing project, wherein the upcoming data processing project has a predetermined time period;
creating a new virtual warehouse associated with the upcoming data processing project; and
deactivating the new virtual warehouse after the predetermined time period.
18. A system comprising:
means for accessing a plurality of remote storage devices collectively storing database data, the means for accessing includes a plurality of virtual warehouses that each share the plurality of remote storage devices, retrieve data from the plurality of remote storage devices, and execute one or more queries received from client devices against the database data, each virtual warehouse including a cache configured to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices; and
means for managing resources coupled to the means for accessing and configured to:
determine predicted data processing requests based on previous patterns of previously received data processing requests from a particular client at a particular time;
monitor data processing requests including received data processing requests and the predicted data processing requests;
monitor resource utilization in the plurality of virtual warehouses;
determine whether additional virtual warehouses are needed based on the data processing requests and the resource utilization in the plurality of virtual warehouses; and
responsive to determining that an additional virtual warehouse is needed, provision a new virtual warehouse that share the plurality of remote storage devices and execute one or more queries received from client devices against the database data.
19. The system of claim 18 , wherein the means for managing resources is further configured to deactivate the new virtual warehouse upon determination that the new virtual warehouse is no longer needed.
20. The system of claim 18 , wherein the means for managing resources is further configured to:
predict a time required to execute a query; and
provision a new virtual warehouse if the time required to execute the query exceeds a threshold value.