Resource allocation for multiple datasets
Systems and methods are disclosed for processing queries against multiple dataset sources. One dataset source can include indexers that index and store data. The system can receive a query that identifies a set of data to be processed and a manner of processing the set of data. The set of data can include a first dataset that is accessible by one or more indexers and a second dataset that is accessible by one or more other dataset sources. A query coordinator can define a query processing scheme for obtaining and processing the set of data that includes a dynamic allocation of multiple layers of partitions. The partitions can operate on multiple worker nodes. The query can then be executed based on the query processing scheme.
1. A method comprising:
receiving, at a search service of a data intake and query system, a query identifying a set of data to be processed and a manner of processing the set of data, the set of data including a first dataset accessible by one or more indexers of the data intake and query system and a second dataset residing in an external data storage system communicatively coupled to the data intake and query system over a network;
defining, by the search service, a query processing scheme for obtaining and processing the set of data, the query processing scheme including:
first instructions to dynamically allocate a first set of processors from a plurality of processors to a first intake layer to obtain data from the one or more indexers and a second set of processors from the plurality of processors to a second intake layer to obtain data from the external data storage system,
second instructions for the one or more indexers to communicate first partial results corresponding to the first dataset to a first subset of the first set of processors,
third instructions directing a second subset of the second set of processors to communicate with the external data storage system to obtain the second dataset from the external data storage system;
fourth instructions directing the second subset of the second set of processors to process the second dataset based on the query processing scheme to obtain second partial results, and
fifth instructions directing the first subset of the first set of processors and the second subset of the second set of processors to combine the first partial results from the one or more indexers and the second partial results from the external data storage system to obtain results of the query and communicate the results of the query to the search service, wherein a quantity of processors for the second subset of the second set of processors is selected based on a determined capability of the external data storage system; and executing the query based on the query processing scheme.
2. The method of claim 1 , wherein the first dataset corresponds to data in a first time period and the second dataset corresponds to data in a second time period.
3. The method of claim 1 , wherein the first dataset corresponds to data stored in one or more hot buckets and the second dataset corresponds to data stored in one or more warm buckets.
4. The method of claim 1 , wherein defining the query processing scheme comprises generating a subquery for the one or more indexers, the subquery identifying the first dataset to be processed and a manner of processing the first dataset to produce the first partial results.
5. The method of claim 1 , wherein the first set of processors comprises multiple processors to obtain the first partial results from the one or more indexers.
6. The method of claim 1 , wherein executing the query comprises monitoring the set of processors.
7. The method of claim 1 , wherein the first partial results comprise raw machine data.
8. The method of claim 1 , wherein the first partial results comprise data derived from raw machine data.
9. The method of claim 1 , wherein the second subset of the second set of processors comprises a processor for each processor supported by the external data storage system.
10. The method of claim 1 , wherein the first partial results include results obtained from a plurality of buckets stored in a shared storage system.
11. The method of claim 1 , wherein the first partial results include results obtained from a plurality of buckets stored in a shared storage system and the first set of processors comprises multiple processors to receive the first partial results from the one or more indexers.
12. The method of claim 1 , wherein defining the query processing scheme comprises generating directed acyclic graph instructions to execute the query using the plurality of processors.
13. The method of claim 1 , wherein defining the query processing scheme comprises generating directed acyclic graph instructions to execute the query using the plurality of processors, and wherein executing the query comprises communicating the directed acyclic graph instructions to the plurality of processors.
14. The method of claim 1 , wherein executing the query comprises receiving the results of the query from the first subset of the first set of processors and the second subset of the second set of processors, processing the results of the query, and communicating the processed results to a client device.
15. The method of claim 1 , wherein executing the query comprises receiving the results of the query from the first subset of the first set of processors and the second subset of the second set of processors, collating the results of the query based on time, and communicating the collated results to a client device.
16. The method of claim 1 , wherein executing the query comprises storing the results of the query from the first subset of the first set of processors and the second subset of the second set of processors in a data store for later use.
17. A computing system, comprising:
one or more processing devices configured to:
receive a query identifying a set of data to be processed and a manner of processing the set of data, the set of data including a first dataset accessible by one or more indexers of a data intake and query system and a second dataset residing in an external data storage system communicatively coupled to the data intake and query system over a network;
define a query processing scheme for obtaining and processing the set of data, the query processing scheme including:
first instructions to dynamically allocate a first set of processors from a plurality of processors to a first intake layer to obtain data from the one or more indexers and a second set of processors from the plurality of processors to a second intake layer to obtain data from the external data storage system,
second instructions for the one or more indexers to communicate first partial results corresponding to the first dataset to a first subset of the first set of processors,
third instructions directing a second subset of the second set of processors to communicate with the external data storage system to obtain the second dataset from the external data storage system;
fourth instructions directing the second subset of the second set of processors to process the second dataset based on the query processing scheme to obtain second partial results, and
fifth instructions directing the first subset of the first set of processors and the second subset of the second set of processors to combine the first partial results from the one or more indexers and the second partial results from the external data storage system to obtain results of the query and communicate the results of the query to the one or more processing devices, wherein a quantity of processors for the second subset of the second set of processors is selected based on a determined capability of the external data storage system; and execute the query based on the query processing scheme.
18. The system of claim 17 , wherein defining the query processing scheme comprises generating a subquery for the one or more indexers, the subquery identifying the first dataset to be processed and a manner of processing the first dataset to produce the first partial results.
19. Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system, cause the computing system to:
receive a query identifying a set of data to be processed and a manner of processing the set of data, the set of data including a first dataset accessible by one or more indexers of a data intake and query system and a second dataset residing in an external data storage system communicatively coupled to the data intake and query system over a network;
define a query processing scheme for obtaining and processing the set of data, the query processing scheme including:
first instructions to dynamically allocate a first set of processors from a plurality of processors to a first intake layer to obtain data from the one or more indexers and a second set of processors from the plurality of processors to a second intake layer to obtain data from the external data storage system,
second instructions for the one or more indexers to communicate first partial results corresponding to the first dataset to a first subset of the first set of processors,
third instructions directing a second subset of the second set of processors to communicate with the external data storage system to obtain the second dataset from the external data storage system;
fourth instructions directing the second subset of the second set of processors to process the second dataset based on the query processing scheme to obtain second partial results, and
fifth instructions directing the first subset of the first set of processors and the second subset of the second set of processors to combine the first partial results from the one or more indexers and the second partial results from the external data storage system to obtain results of the query and communicate the results of the query to the computing system, wherein a quantity of processors for the second subset of the second set of processors is selected based on a determined capability of the external data storage system; and execute the query based on the query processing scheme.