Systems and methods for scaling data warehouses
Example resource provisioning systems and methods are described. In one implementation, multiple processing resources are provided within a data warehouse. The processing resources include at least one processor and at least one storage device. At least one query to process database data is received. At least some of the processing resources may process the database data. When a processing capacity of the processing resources has reached a threshold processing capacity, the processing capacity is automatically scaled by adding at least one additional processor to the data warehouse.
1. A method comprising:
provisioning a first data warehouse comprising a plurality of processing resources, the plurality of processing resources comprising at least one processor and at least one storage device;
receiving a processing request to process database data stored on a storage platform comprising a plurality of shared storage devices in association with the first data warehouse;
determining that a processing capacity of the plurality of processing resources of the first data warehouse would reach a threshold processing capacity when processing the processing request based on metadata including information regarding pre-cached data on the plurality of processing resources of the first data warehouse; and
provisioning a second data warehouse to process at least a portion of the processing request after determining that the processing capacity of the plurality of processing resources of the first data warehouse would reach the threshold processing capacity when processing the processing request.
2. The method of claim 1 , further comprising:
scaling one or more of the processing capacity of the plurality of processing resources of the first data warehouse by adding additional processing resources to the first data warehouse and the storage capacity of the storage platform by adding at least one additional storage device.
3. The method of claim 2 , further comprising:
determining that the at least one additional processor is no longer needed; and
removing the at least one additional processor from the first data warehouse.
4. The method of claim 2 , wherein adding additional processing resources comprises adding at least one additional processor to process the at least one request to process database data stored in the first data warehouse.
5. The method of claim 2 , wherein adding at least one additional processor to the first data warehouse is accomplished without affecting a storage capacity of the plurality of shared storage devices.
6. The method of claim 1 , wherein determining that the processing capacity would reach a threshold processing capacity comprises determining that a current query processing delay would exceed a threshold delay.
7. The method of claim 1 , wherein each processing resource comprises a processor and a storage device.
8. The method of claim 1 , wherein the storage platform is physically separate from the plurality of processing resources of the first data warehouse.
9. The method of claim 1 , wherein the processing request comprises a data retrieval request or a data storage request.
10. A system, comprising:
a processing device having a hardware processor executing instructions programmed to:
provision a first data warehouse comprising a plurality of processing resources, the plurality of processing resources comprising at least one processor and at least one storage device;
receive a processing request to process database data stored on a storage platform comprising a plurality of shared storage devices in association with the first data warehouse;
determine that a processing capacity of the plurality of processing resources of the first data warehouse would reach a threshold processing capacity when processing the processing request based on metadata including information regarding pre-cached data on the plurality of processing resources of the first data warehouse; and
provision a second data warehouse to process at least a portion of the processing request after determining that the processing capacity of the plurality of processing resources of the first data warehouse would reach the threshold processing capacity when processing the processing request.
11. The system of claim 10 , wherein the processing device is further programmed to:
scale one or more of the processing capacity of the plurality of processing resources of the first data warehouse by adding additional processing resources to the first data warehouse and the storage capacity of the storage platform by adding at least one additional storage device.
12. The system of claim 11 , wherein the processing device is further programmed to:
determine that the at least one additional processor is no longer needed; and
remove the at least one additional processor from the first data warehouse.
13. The system of claim 11 , wherein the processing device is further programmed to:
add at least one additional processor to process the at least one request to process database data stored in the first data warehouse.
14. The system of claim 11 , wherein adding at least one additional processor to the first data warehouse is accomplished without affecting a storage capacity of the plurality of shared storage devices.
15. The system of claim 10 , wherein determining that the processing capacity would reach a threshold processing capacity comprises determining that a current query processing delay would exceed a threshold delay.
16. The system of claim 10 , wherein each processing resource comprises a processor and a storage device.
17. The system of claim 10 , wherein the storage platform is physically separate from the plurality of processing resources of the first data warehouse.
18. The system of claim 10 , wherein the processing request comprises a data retrieval request or a data storage request.
19. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to:
provision a first data warehouse comprising a plurality of processing resources, the plurality of processing resources comprising at least one processor and at least one storage device;
receive a processing request to process database data stored on a storage platform comprising a plurality of shared storage devices in association with the first data warehouse;
determine that a processing capacity of the plurality of processing resources of the first data warehouse would reach a threshold processing capacity when processing the processing request based on metadata including information regarding pre-cached data on the plurality of processing resources of the first data warehouse; and
provision a second data warehouse to process at least a portion of the processing request after determining that the processing capacity of the plurality of processing resources of the first data warehouse would reach the threshold processing capacity when processing the processing request.
20. The non-transitory computer-readable medium of claim 19 , wherein the instructions further cause the processor to:
scale one or more of the processing capacity of the plurality of processing resources of the first data warehouse by adding additional processing resources to the first data warehouse and the storage capacity of the storage platform by adding at least one additional storage device.
21. The non-transitory computer-readable medium of claim 20 , wherein the instructions further cause the processor to:
determine that the at least one additional processor is no longer needed; and
remove the at least one additional processor from the first data warehouse.
22. The non-transitory computer-readable medium of claim 20 , wherein the instructions further cause the processor to:
add at least one additional processor to process the at least one request to process database data stored in the first data warehouse.
23. The non-transitory computer-readable medium of claim 20 , wherein adding at least one additional processor to the first data warehouse is accomplished without affecting a storage capacity of the plurality of shared storage devices.
24. The non-transitory computer-readable medium of claim 19 , wherein determining that the processing capacity would reach a threshold processing capacity comprises determining that a current query processing delay would exceed a threshold delay.
25. The non-transitory computer-readable medium of claim 19 , wherein each processing resource comprises a processor and a storage device.
26. The non-transitory computer-readable medium of claim 19 , wherein the storage platform is physically separate from the plurality of processing resources of the first data warehouse.
27. The non-transitory computer-readable medium of claim 19 , wherein the processing request comprises a data retrieval request or a data storage request.