Resource management systems and methods
Example resource management systems and methods are described. In one implementation, a system includes a memory and a processing device operatively coupled to the memory. The processing device is to receive a query referencing database data stored in a storage platform, determine a task associated with processing the received query, and create an execution node comprising cache resources and processing resources. Furthermore, a size of the cache resources of the execution node is determined upon creation of the execution node, based at least in part on the task, and processing resources of the execution node are determined upon creation of the execution node, based at least in part on the task. The execution node is included within a plurality of execution nodes to process the task associated with processing the received query.
1. A system comprising:
a memory; and
a processing device operatively coupled to the memory, the processing device to:
receive a query referencing database data stored in a storage platform;
determine a task associated with processing the received query;
create, by the processing device, an execution node comprising cache resources and processing resources, wherein:
a size of the cache resources of the execution node is determined upon creation of the execution node, based at least in part on the task; and
processing resources of the execution node are determined upon creation of the execution node, based at least in part on the task; and
include the execution node within a plurality of execution nodes to process the task associated with processing the received query.
2. The system of claim 1 , wherein the storage platform comprises a plurality of storage devices, and
wherein each of the plurality of execution nodes is communicatively coupled to a subset of the plurality of storage devices.
3. The system of claim 1 , wherein creating the new execution node is performed responsive to determining a deficit in at least one of processing resources or caching resources.
4. The system of claim 1 , wherein the processing device is further to determine portions of the database data that are cached within the respective caches of the plurality of execution nodes.
5. The system of claim 1 , wherein the at least one task comprises a first task and a second task, and
wherein the processing device is further to:
allocate the first task of the received query to a first execution node of the plurality of execution nodes responsive to determining that the first task references a first portion of the database data that is stored in the cache of the first execution node; and
allocate the second task of the received query to the new execution node.
6. The system of claim 1 , wherein a size of the cache of the new execution node is dynamically determined upon creation of the new execution node, based at least in part on the least one task associated with processing the received query to be processed by the new execution node.
7. The system of claim 1 , wherein the respective caches of multiple ones of the plurality of execution nodes cache a same portion of the database data at a same time.
8. A method comprising:
receiving a query referencing database data stored in a storage platform;
determining a task associated with processing the received query;
creating, by a processing device, an execution node comprising cache resources and processing resources, wherein:
a size of the cache resources of the execution node is determined upon creation of the execution node, based at least in part on the task; and
processing resources of the execution node are determined upon creation of the execution node, based at least in part on the task; and
including the execution node within a plurality of execution nodes to process the task associated with processing the received query.
9. The method of claim 8 , wherein the storage platform comprises a plurality of storage devices, and
wherein each of the plurality of execution nodes is communicatively coupled to a subset of the plurality of storage devices.
10. The method of claim 8 , wherein creating the new execution node is performed responsive to determining a deficit in at least one of processing resources or caching resources.
11. The method of claim 8 , further comprising determining portions of the database data that are cached within the respective caches of the plurality of execution nodes.
12. The method of claim 8 , wherein the at least one task comprises a first task and a second task, and
wherein method further comprises:
allocating the first task of the received query to a first execution node of the plurality of execution nodes responsive to determining that the first task references a first portion of the database data that is stored in the cache of the first execution node; and
allocating the second task of the received query to the new execution node.
13. The method of claim 8 , wherein a size of the cache of the new execution node is dynamically determined upon creation of the new execution node, based at least in part on the least one task associated with processing the received query to be processed by the new execution node.
14. The method of claim 8 , wherein the respective caches of multiple ones of the plurality of execution nodes cache a same portion of the database data at a same time.
15. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processing device, cause the processing device to:
receive a query referencing database data stored in a storage platform;
determine a task associated with processing the received query;
create an execution node comprising cache resources and processing resources, wherein:
a size of the cache resources of the execution node is determined upon creation of the execution node, based at least in part on the task; and
processing resources of the execution node are determined upon creation of the execution node, based at least in part on the task; and
include the execution node within a plurality of execution nodes to process the task associated with processing the received query.
16. The non-transitory computer-readable medium of claim 15 , wherein the storage platform comprises a plurality of storage devices, and
wherein each of the plurality of execution nodes is communicatively coupled to a subset of the plurality of storage devices.
17. The non-transitory computer-readable medium of claim 15 , wherein creating the new execution node is performed responsive to determining a deficit in at least one of processing resources or caching resources.
18. The non-transitory computer-readable medium of claim 15 , wherein the at least one task comprises a first task and a second task, and
wherein the processing device is further to:
allocate the first task of the received query to a first execution node of the plurality of execution nodes responsive to determining that the first task references a first portion of the database data that is stored in the cache of the first execution node; and
allocate the second task of the received query to the new execution node.
19. The non-transitory computer-readable medium of claim 15 , wherein a size of the cache of the new execution node is dynamically determined upon creation of the new execution node, based at least in part on the least one task associated with processing the received query to be processed by the new execution node.
20. The non-transitory computer-readable medium of claim 15 , wherein the respective caches of multiple ones of the plurality of execution nodes cache a same portion of the database data at a same time.