Resource management systems and methods
Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.
1. A method, comprising:
receiving, by a resource manager running on a first executor node of a plurality of executor nodes, a query to process database data, wherein each of the plurality of executor nodes comprises virtual resources including at least one CPU and a cache to cache data from a plurality of storage devices;
retrieving, by the resource manager, information regarding storage of at least a portion of the database data to one or more caches accessible to a plurality of execution nodes; and
allocating, based at least in part on the retrieved information, the query to one or more of the plurality of executor nodes with access to one or more caches storing at least a portion of the database data, the one or more execution nodes to process the database data based at least in part on an organization of the database data within the plurality of cache memories of the plurality of executor nodes and the plurality of storage devices.
2. The method of claim 1 , further comprising determining an organization of the database data within a plurality of cache memories of the plurality of executor nodes based on metadata associated with the database data.
3. The method of claim 1 , wherein allocating the query comprises:
determining more than one task required to process the query; and
allocating the more than one task to the one or more executor nodes.
4. The method of claim 3 , wherein allocating the query to one or more of the plurality of executor nodes further comprises:
determining which executor nodes of the plurality of executor nodes already cache at least a portion of the database data needed to process the query in a respective cache based on the organization of the database data within a plurality of caches of the plurality of executor nodes and the plurality of storage devices; and
allocating the query to one or more of the executor nodes that already cache at least a portion of the database data.
5. The method of claim 4 , wherein portions of the database data are allocated within one or more of a plurality of caches of the plurality of executor nodes based on how frequently the database data is accessed.
6. The method of claim 1 , wherein the plurality of executor nodes is organized into one or more virtual warehouses, each of the one or more virtual warehouses to dynamically communicate with a subset of the plurality of storage devices based on the query.
7. The method of claim 6 , wherein the one or more virtual warehouses are logically mapped with each other.
8. The method of claim 1 , wherein the plurality of executor nodes is external to and separate from the plurality of storage devices.
9. The method of claim 1 , wherein the plurality of storage devices comprises a virtual database.
10. A system, comprising:
a memory; and
one or more processors operatively coupled to the memory, the one or more processors to:
receive, by a resource manager running on a first executor node of a plurality of executor nodes, a query to process database data, wherein each of the plurality of executor nodes comprises virtual resources including at least one CPU core and a cache to cache data from a plurality of storage devices;
retrieve, by the resource manager, information regarding storage of at least a portion of the database data to one or more caches accessible to a plurality of execution nodes; and
allocate, based at least in part on the retrieved information, the query to one or more of the plurality of executor nodes with access to one or more caches storing at least a portion of the database data, the one or more execution nodes to process the database data based at least in part on an organization of the database data within the plurality of cache memories of the plurality of executor nodes and the plurality of storage devices.
11. The system of claim 10 , wherein the one or more processors are further to determine an organization of the database data within a plurality of cache memories of the plurality of executor nodes based on metadata associated with the database data.
12. The system of claim 10 , wherein to allocate the query the one or more processors are to:
determine more than one task required to process the query; and
allocate the more than one task to the one or more executor nodes.
13. The system of claim 12 , wherein to allocate the query to one or more of the plurality of executor nodes, the one or more processors are further to:
determine which executor nodes of the plurality of executor nodes already cache at least a portion of the database data needed to process the query in a respective cache based on the organization of the database data within a plurality of caches of the plurality of executor nodes and the plurality of storage devices; and
allocate the query to one or more of the executor nodes that already cache at least a portion of the database data.
14. The system of claim 13 , wherein portions of the database data are allocated within one or more of a plurality of caches of the plurality of executor nodes based on how frequently the database data is accessed.
15. The system of claim 10 , wherein the plurality of executor nodes is organized into one or more virtual warehouses, each of the one or more virtual warehouses to dynamically communicate with a subset of the plurality of storage devices based on the query.
16. The system of claim 15 , wherein the one or more virtual warehouses are logically mapped with each other.
17. The system of claim 10 , wherein the plurality of executor nodes is external to and separate from the plurality of storage devices.
18. The system of claim 10 , wherein the plurality of storage devices comprises a virtual database.
19. A non-transitory computer-readable medium having instructions stored thereon which, when executed by one or more processors, causes the one or more processors to:
receive, by a resource manager running on a first executor node of a plurality of executor nodes, a query to process database data, wherein each of the plurality of executor nodes comprises virtual resources including at least one CPU core and a cache to cache data from a plurality of storage devices;
retrieve, by the resource manager, information regarding storage of at least a portion of the database data to one or more caches accessible to a plurality of execution nodes; and
allocate, based at least in part on the retrieved information, the query to one or more of the plurality of executor nodes with access to one or more caches storing at least a portion of the database data, the one or more execution nodes to process the database data based at least in part on an organization of the database data within the plurality of cache memories of the plurality of executor nodes and the plurality of storage devices.
20. The non-transitory computer-readable medium of claim 19 , wherein the one or more processors are further to determine an organization of the database data within a plurality of cache memories of the plurality of executor nodes based on metadata associated with the database data.
21. The non-transitory computer-readable medium of claim 19 , wherein to allocate the query the one or more processors are to:
determine more than one task required to process the query; and
allocate the more than one task to the one or more executor nodes.
22. The non-transitory computer-readable medium of claim 21 , wherein to allocate the query to one or more of the plurality of executor nodes, the one or more processors are further to:
determine which executor nodes of the plurality of executor nodes already cache at least a portion of the database data needed to process the query in a respective cache based on the organization of the database data within a plurality of caches of the plurality of executor nodes and the plurality of storage devices; and
allocate the query to one or more of the executor nodes that already cache at least a portion of the database data.
23. The non-transitory computer-readable medium of claim 22 , wherein portions of the database data are allocated within one or more of a plurality of caches of the plurality of executor nodes based on how frequently the database data is accessed.
24. The non-transitory computer-readable medium of claim 19 , wherein the plurality of executor nodes is organized into one or more virtual warehouses, each of the one or more virtual warehouses to dynamically communicate with a subset of the plurality of storage devices based on the query.
25. The non-transitory computer-readable medium of claim 24 , wherein the one or more virtual warehouses are logically mapped with each other.
26. The non-transitory computer-readable medium of claim 19 , wherein the plurality of executor nodes is external to and separate from the plurality of storage devices.
27. The non-transitory computer-readable medium of claim 19 , wherein the plurality of storage devices comprises a virtual database.