Resource management systems and methods
Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.
1. A method, comprising:
storing at least a portion of database data to one or more caches accessible to a plurality of execution nodes;
receiving, by a resource manager running on a first execution node of the plurality of execution nodes, a query to process the database data, wherein each of the plurality of execution nodes comprises at least one processor and a cache to cache data from a plurality of storage devices to independently process an allocated portion of the database data, wherein the resource manager is scaled independently from other resource managers based on the query and associated database data to be processed, wherein the cache varies in size among the plurality of execution nodes; and
allocating, the query to one or more of the plurality of execution nodes to process the database data, the one or more of the plurality of execution nodes having access to the one or more caches storing at least a portion of the database data, wherein the one or more of the plurality of execution nodes process the database data based at least in part on an organization of the database data within the one or more caches of the plurality of execution nodes and the plurality of storage devices.
2. The method of claim 1 , wherein the resource manager comprises a virtual warehouse manager and wherein the plurality of execution nodes belong to one or more virtual warehouses managed by the virtual warehouse manager.
3. The method of claim 2 , wherein the virtual warehouse manager comprises at least a logical mapping of the one or more virtual warehouses, wherein the logical mapping is based on access control parameters related to security and resource access management settings.
4. The method of claim 1 , further comprising:
retrieving, by the resource manager, information regarding the storage of the at least a portion of the database data to one or more caches accessible to the plurality of execution nodes, wherein the retrieving comprises:
identifying candidate execution nodes for processing the received query, wherein the candidate execution nodes have already cached the portion of the database data needed to process the query, wherein the query is allocated based at least in part on the retrieved information; and
retrieving the information from the candidate execution nodes.
5. The method of claim 1 , wherein the organization of the database data is stored in a structured and optimized format, and wherein the database data is accessible as a service separate from computing or processing resources.
6. The method of claim 1 , wherein the query is in a form of one or more structured query language (SQL) statements.
7. The method of claim 1 , wherein the at least one processor is a one of a physical processor or a virtual processor.
8. A system comprising:
a memory; and
a processing device operably coupled to the memory, the processing device to:
storing at least a portion of database data to one or more caches accessible to a plurality of execution nodes;
receive, in a resource manager running on a first execution node of the plurality of execution nodes, a query to process the database data, wherein each of the plurality of execution nodes comprises at least one processor and a cache to cache data from a plurality of storage devices to independently process an allocated portion of the database data, wherein the resource manager is scaled independently from other resource managers based on the query and associated database data to be processed, wherein the cache varies in size among the plurality of execution nodes; and
allocate, the query to one or more of the plurality of execution nodes to process the database data, the one or more of the plurality of execution nodes having access to the one or more caches storing at least a portion of the database data, wherein the one or more of the plurality of execution nodes process the database data based at least in part on an organization of the database data within the one or more caches of the plurality of execution nodes and the plurality of storage devices.
9. The system of claim 8 , wherein the resource manager comprises a virtual warehouse manager and wherein the plurality of execution nodes belong to one or more virtual warehouses managed by the virtual warehouse manager.
10. The system of claim 9 , wherein the virtual warehouse manager comprises at least a logical mapping of the one or more virtual warehouses, wherein the logical mapping is based on access control parameters related to security and resource access management settings.
11. The system of claim 8 , wherein the processing device is to retrieve information regarding the storage of a portion of the database data to one or more caches accessible to the plurality of execution nodes by:
identifying candidate execution nodes for processing the received query, wherein the candidate execution nodes have already cached the portion of the database data needed to process the query, and wherein the query is allocated based at least in part on the retrieved information; and
retrieving the information from the candidate execution nodes.
12. The system of claim 8 , wherein the organization of the database data is stored in a structured and optimized format, and wherein the database data is accessible as a service separate from computing or processing resources.
13. The system of claim 8 , wherein the query is in a form of one or more structured query language (SQL) statements.
14. The system of claim 8 , wherein the at least one processor is a one of a physical processor or a virtual processor.
15. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a processing device, cause the processing device to:
storing at least a portion of database data to one or more caches accessible to a plurality of execution nodes;
receive, in a resource manager running on a first execution node of the plurality of execution nodes, a query to process the database data, wherein each of the plurality of execution nodes comprises at least one processor and a cache to cache data from a plurality of storage devices to independently process an allocated portion of the database data, wherein the resource manager is scaled independently from other resource managers based on the query and associated database data to be processed, wherein the cache varies in size among the plurality of execution nodes; and
allocate, the query to one or more of the plurality of execution nodes to process the database data, the one or more of the plurality of execution nodes having access to the one or more caches storing at least a portion of the database data, wherein the one or more of the plurality of execution nodes process the database data based at least in part on an organization of the database data within the one or more caches of the plurality of execution nodes and the plurality of storage devices.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the resource manager comprises a virtual warehouse manager and wherein the plurality of execution nodes belong to one or more virtual warehouses managed by the virtual warehouse manager.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the virtual warehouse manager comprises at least a logical mapping of the one or more virtual warehouses, wherein the logical mapping is based on access control parameters related to security and resource access management settings.
18. The non-transitory computer-readable storage medium of claim 15 , further comprising instructions stored thereon that, when executed by the processing device, cause the processing device to:
identify candidate execution nodes for processing the received query, wherein the candidate execution nodes have already cached the portion of the database data needed to process the query; and
retrieve information from the candidate execution nodes.
19. The non-transitory computer-readable storage medium of claim 15 , wherein the organization of the database data is stored in a structured and optimized format, and wherein the database data is accessible as a service separate from computing or processing resources.
20. The non-transitory computer-readable storage medium of claim 15 , wherein the at least one processor is a one of a physical processor or a virtual processor.