Resource management systems and methods
Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.
1. A virtual data warehouse management system having one or more processors, comprising:
a database manager executable by the one or more processors, the database manager to receive a query to process database data stored in a storage platform comprising one or more storage devices; and
a plurality of processors operatively coupled to the database manager, wherein:
each processor of the plurality of processors is associated with a stateless node and associated with a cache resource comprising at least one cache memory to cache database data from the one or more storage devices, and
the database manager further to:
determine an availability of one or more resources of at least one of the plurality of processors to process the database data;
allocate work to the plurality of processors to process the database data responsive to determining the availability of the one or more resources of the at least one of the plurality of processors;
detect a failure associated with the stateless node to process the database data; and
replace the stateless node with a different node without recreating a particular state.
2. The virtual data warehouse management system of claim 1 , further comprising a metadata manager to determine the organization of the database data within the plurality of cache memories based on metadata associated with the database data.
3. The virtual data warehouse management system of claim 2 , wherein to allocate the work, the database manager determines which processors of the plurality of processors already cache at least a portion of the database data needed to process the query in an associated cache memory based on the organization of the database data within the plurality of cache memories.
4. The virtual data warehouse management system of claim 1 , wherein the database data is allocated within one or more of the one or more storage devices associated with the plurality of cache memories based on how frequently the database data is accessed.
5. The virtual data warehouse management system of claim 1 , wherein the database manager to allocate the work to a plurality of virtual data warehouse instances to process the query.
6. The virtual data warehouse management system of claim 2 , wherein the plurality of processors is coupled to the one or more storage devices, which are external to and separate from the plurality of processors.
7. The virtual data warehouse management system of claim 6 , wherein the one or more storage devices comprise a virtual database.
8. The virtual data warehouse management system of claim 1 , wherein the database manager is external to and separate from computing resources of the plurality of processors.
9. The virtual data warehouse management system of claim 1 , wherein the one or more storage devices comprise a plurality of virtual data stores.
10. The virtual data warehouse management system of claim 9 , wherein the one or more storage devices comprise a logical mapping between the plurality of virtual data stores.
11. A method, comprising:
providing a plurality of processors and a plurality of cache memories in association with a virtual data warehouse management system, wherein each of the plurality of processors is associated with a stateless node and associated with a cache resource comprising at least one cache memory to cache database data from one or more storage devices;
receiving a query to process database data stored in a storage platform comprising the one or more storage devices, wherein the plurality of processors is to process the database data;
determining an availability of one or more resources of at least one of the plurality of processors to process the database data;
allocating work to the plurality of processors to process the database data responsive to determining the availability of the one or more resources of the at least one of the plurality of processors; and
detecting a failure associated with the stateless node to process the database data; and
replacing the stateless node with a different node without recreating a particular state.
12. The method of claim 11 , further comprising determining the organization of the database data within the plurality of cache memories and the one or more data stores based on metadata associated with the database data.
13. The method of claim 12 , wherein allocating the work to the plurality of processors comprises determining which processors of the plurality of processors already cache at least a portion of the database data needed to process the query in an associated cache memory based on the organization of the database data within the plurality of cache memories.
14. The method of claim 11 , wherein the database data is allocated within one or more of the plurality of cache memories associated with the plurality of processors based on how frequently the database data is accessed.
15. The method of claim 11 , further comprising allocating the work to a plurality of virtual data warehouse instances to process the query.
16. The method of claim 11 , wherein the plurality of processors is coupled to the one or more storage devices, and wherein the one or more storage devices are external to and separate from the plurality of processors.
17. The method of claim 16 , wherein the one or more storage devices comprise a virtual database.
18. The method of claim 11 , wherein the allocating work to the plurality of processors is performed by a database manager that is external to and separate from computing resources of the plurality of processors.
19. The method of claim 11 , wherein the one or more storage devices comprise a plurality of virtual data stores.
20. The method of claim 19 , wherein the one or more storage devices comprise a logical mapping between the plurality of virtual data stores.
21. A non-transitory computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to:
provide a plurality of processors in association with a virtual data warehouse management system, wherein each of the plurality of processors is associated with a stateless node and associated with a cache resource comprising at least one cache memory to cache database data from one or more storage devices;
receive a query to process database data stored in a storage platform comprising the one or more storage devices, wherein each of the plurality of processors to process the database data;
determine an availability of one or more resources of at least one of the plurality of processors to process the database data;
allocate work to the plurality of processors to process the database data responsive to determining the availability of the one or more resources of the at least one of the plurality of processors;
detect a failure associated with the stateless node to process the database data; and
replace the stateless node with a different node without recreating a particular state.
22. The non-transitory computer-readable medium of claim 21 , wherein the processor is further to determine the organization of the database data within the plurality of cache memories based on metadata associated with the database data.
23. The non-transitory computer-readable medium of claim 22 , wherein to allocate the work to the plurality of processors, the processor is further to determine which processors of the plurality of processors already cache at least a portion of the database data needed to process the query in their associated cache memory based on the organization of the database data within the plurality of cache memories.
24. The non-transitory computer-readable medium of claim 21 , wherein the database data is allocated within one or more of the plurality of cache memories associated with the plurality of processors based on how frequently the database data is accessed.
25. The non-transitory computer-readable medium of claim 21 , wherein the processor is to allocate the work to a plurality of virtual data warehouse instances to process the query.
26. The non-transitory computer-readable medium of claim 21 , wherein the plurality of processors is coupled to the one or more storage devices, and wherein the one or more storage devices are external to and separate from the plurality of processors.
27. The non-transitory computer-readable medium of claim 26 , wherein the one or more storage devices comprise a virtual database.
28. The non-transitory computer-readable medium of claim 21 , wherein to allocate the work to the plurality of processors, the processor uses a database manager that is external to and separate from computing resources of the plurality of processors.
29. The non-transitory computer-readable medium of claim 21 , wherein the one or more storage devices comprise a plurality of virtual data stores.
30. The non-transitory computer-readable medium of claim 29 , wherein the one or more storage devices comprise a logical mapping between the plurality of virtual data stores.