Resource management systems and methods
Example resource management systems and methods are described. In one implementation, a system includes a memory and a processing device operatively coupled to the memory. The processing device is to: provide a plurality of execution nodes, each execution node comprising a cache and a processor, each execution node communicatively coupled to a storage platform; determine at least one task associated with processing a received query referencing database data stored in the storage platform; and create a new execution node within the plurality of execution nodes to process the at least one task associated with processing the received query.
1. A system comprising:
a memory; and
a processing device operatively coupled to the memory, the processing device to:
provide a plurality of execution nodes, each execution node comprising a cache and a processor, each execution node communicatively coupled to a storage platform;
after providing the plurality of execution nodes, receive a query referencing database data stored in the storage platform;
determine a task associated with processing the received query;
create a new execution node comprising cache resources and processing resources, wherein a size of the cache resources of the new execution node is determined upon creation of the new execution node, based at least in part on the task; and
include the new execution node within the plurality of execution nodes to process the task associated with processing the received query, wherein the processing resources of the new execution node are determined when the new execution node is created, based on the task to be processed by the plurality of execution nodes including the new execution node.
2. The system of claim 1 , wherein the storage platform comprises a plurality of storage devices, and
wherein each of the plurality of execution nodes is communicatively coupled to a subset of the plurality of storage devices.
3. The system of claim 1 , wherein creating the new execution node is performed responsive to determining a deficit in at least one of processing resources or caching resources.
4. The system of claim 1 , wherein the processing device is further to determine portions of the database data that are cached within the respective caches of the plurality of execution nodes.
5. The system of claim 1 , wherein the task comprises a first task and a second task, and
wherein the processing device is further to:
allocate the first task of the received query to a first execution node of the plurality of execution nodes responsive to determining that the first task references a first portion of the database data that is stored in the cache of the first execution node; and
allocate the second task of the received query to the new execution node.
6. The system of claim 1 , wherein the respective caches of multiple ones of the plurality of execution nodes cache a same portion of the database data at a same time.
7. A method comprising:
providing a plurality of execution nodes, each execution node comprising a cache and a processor, each execution node communicatively coupled to a storage platform;
after providing the plurality of execution nodes, receive a query referencing database data stored in the storage platform;
determining a task associated with processing the received query;
creating, by a processing device, a new execution node having cache resources and processing resources, wherein a size of the cache resources of the new execution node is determined upon creation of the new execution node, based at least in part on the task; and
include the new execution node within the plurality of execution nodes to process the task associated with processing the received query, wherein the processing resources of the new execution node are determined when the new execution node is created, based on the task to be processed by the plurality of execution nodes including the new execution node.
8. The method of claim 7 , wherein the storage platform comprises a plurality of storage devices, and
wherein each of the plurality of execution nodes is communicatively coupled to a subset of the plurality of storage devices.
9. The method of claim 7 , wherein creating the new execution node is performed responsive to determining a deficit in at least one of processing resources or caching resources.
10. The method of claim 7 , further comprising determining portions of the database data that are cached within the respective caches of the plurality of execution nodes.
11. The method of claim 7 , wherein the task comprises a first task and a second task, and
wherein the method further comprises:
allocating the first task of the received query to a first execution node of the plurality of execution nodes responsive to determining that the first task references a first portion of the database data that is stored in the cache of the first execution node; and
allocating the second task of the received query to the new execution node.
12. The method of claim 7 , wherein the respective caches of multiple ones of the plurality of execution nodes cache a same portion of the database data at a same time.
13. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processing device, cause the processing device to:
provide a plurality of execution nodes, each execution node comprising a cache and a processor, each execution node communicatively coupled to a storage platform;
after providing the plurality of execution nodes, receive a query referencing database data stored in the storage platform;
determine a task associated with processing the received query;
create, by the processing device, a new execution node comprising cache resources and processing resources, wherein a size of the cache resources of the new execution node is determined upon creation of the new execution node, based at least in part on the task; and
include the new execution node within the plurality of execution nodes to process the task associated with processing the received query, wherein the processing resources of the new execution node are determined when the new execution node is created, based on the task to be processed by plurality of execution nodes including the new execution node.
14. The non-transitory computer-readable medium of claim 13 , wherein the storage platform comprises a plurality of storage devices, and
wherein each of the plurality of execution nodes is communicatively coupled to a subset of the plurality of storage devices.
15. The non-transitory computer-readable medium of claim 13 , wherein creating the new execution node is performed responsive to determining a deficit in at least one of processing resources or caching resources.
16. The non-transitory computer-readable medium of claim 13 , wherein the task comprises a first task and a second task, and
wherein the processing device is further to:
allocate the first task of the received query to a first execution node of the plurality of execution nodes responsive to determining that the first task references a first portion of the database data that is stored in the cache of the first execution node; and
allocate the second task of the received query to the new execution node.
17. The non-transitory computer-readable medium of claim 13 , wherein the respective caches of multiple ones of the plurality of execution nodes cache a same portion of the database data at a same time.