Multi-layer partition allocation for query execution
Systems and methods are disclosed for processing and executing queries against one or more dataset sources, where the queries identify a set of data to be processed and a manner of processing the set of data. To query the dataset sources, a query coordinator generates a query processing scheme that includes a dynamic allocation of multiple layers of partitions. The query is then executed based on the query processing scheme.
1. A method comprising:
receiving, by a data intake and query system, a query identifying a set of data to be processed and a manner of processing the set of data;
defining, by the data intake and query system, a query processing scheme for obtaining and processing the set of data;
dynamically allocating a plurality of processors to multiple layers of processors to execute the query based on defining the query processing scheme, wherein dynamically allocating the plurality of processors to multiple layers of processors to execute the query comprises:
dynamically allocating a first subset of the plurality of processors to a first layer of processors to obtain, from one or more dataset sources, the set of data based on the query identifying the set of data to be processed, and
dynamically allocating a second subset of the plurality of processors to a second layer of processors to process the set of data based on the query identifying the manner of processing the set of data, wherein each of the first layer of processors and the second layer of processors can obtain the set of data and process the set of data; and
executing the query based on the query processing scheme.
2. The method of claim 1 , wherein the first layer of processors comprises an intake layer including the first subset of the plurality of processors to obtain the set of data from at least one dataset source of the one or more dataset sources.
3. The method of claim 1 , wherein the first layer of processors comprises an intake layer including the first subset of the plurality of processors to obtain the set of data from at least one dataset source of the one or more dataset sources and a third layer of processors comprises a storage layer including at least one processor to communicate data to a dataset destination.
4. The method of claim 1 , wherein the first layer of processors comprises an intake layer including the first subset of the plurality of processors to obtain the set of data from at least one dataset source of the one or more dataset sources and a third layer of processors comprises a branch layer including at least one processor to communicate data to multiple locations.
5. The method of claim 1 , wherein the first layer of processors comprises an intake layer including the first subset of the plurality of processors to obtain the set of data from at least one dataset source of the one or more dataset sources, the second layer of processors comprises a processing layer including the second subset of the plurality of processors to receive and process the set of data from the intake layer, and a third layer of processors comprises a collector layer including a third subset of the plurality of processors to collect data from the processing layer.
6. The method of claim 1 , wherein dynamically allocating the first subset of the plurality of processors to the first layer of processors to obtain, from the one or more dataset sources, the set of data is further based on a determined processing capability of a dataset source of the one or more dataset sources.
7. The method of claim 1 , wherein the first layer of processors comprises an intake layer, wherein dynamically allocating the first subset of the plurality of processors to the first layer of processors to obtain, from the one or more dataset sources, the set of data, comprises dynamically allocating a particular portion of the plurality of processors to the intake layer for each of the one or more dataset sources.
8. The method of claim 1 , wherein defining the query processing scheme comprises determining a processing capability of a dataset source of the one or more dataset sources, wherein dynamically allocating the plurality of processors to the multiple layers of processors is based on the determining the processing capability.
9. The method of claim 1 , wherein defining the query processing scheme comprises determining a processing capability of a dataset source of the one or more dataset sources and generating a subquery for the dataset source based on the determining the processing capability, the subquery identifying at least a portion of the set of data to be processed and a manner of processing the at least a portion of the set of data.
10. The method of claim 1 , wherein defining the query processing scheme comprises generating instructions for execution by the plurality of processors, and wherein executing the query comprises communicating the instructions to the plurality of processors.
11. The method of claim 1 , wherein the one or more dataset sources comprise an external data source.
12. The method of claim 1 , wherein the one or more dataset sources comprise an external data source and the first layer of processors comprises an intake layer including the first subset of the plurality of processors to obtain the set of data from the external data source.
13. The method of claim 1 , wherein the one or more dataset sources comprise an external data source and the first layer of processors comprises an intake layer including a processor for each processor supported by the external data source.
14. The method of claim 1 , wherein the one or more dataset sources comprise a plurality of buckets from a common storage.
15. The method of claim 1 , wherein the one or more dataset sources comprise a plurality of buckets from a common storage and the first layer of processors comprises an intake layer including the first subset of the plurality of processors to obtain the set of data from the plurality of buckets.
16. The method of claim 1 , wherein the one or more dataset sources comprise a plurality of buckets from a common storage and the first layer of processors comprises an intake layer including the first subset of the plurality of processors to obtain the set of data from the plurality of buckets based on a determined quantity of available processors.
17. The method of claim 1 , wherein the one or more dataset sources comprise a plurality of indexers and the first layer of processors comprises an intake layer including the first subset of the plurality of processors to obtain the set of data from the plurality of indexers.
18. The method of claim 1 , wherein the one or more dataset sources comprise a plurality of indexers and the first layer of processors comprises an intake layer including the first subset of the plurality of processors to obtain the set of data from the plurality of indexers based on a determined quantity of available processors.
19. The method of claim 1 , wherein the one or more dataset sources comprise an ingested data buffer.
20. The method of claim 1 , wherein the one or more dataset sources comprise an ingested data buffer and the first layer of processors comprises an intake layer including the first subset of the plurality of processors to obtain the set of data from the ingested data buffer.
21. The method of claim 1 , wherein the one or more dataset sources comprise an ingested data buffer and the first layer of processors comprises an intake layer including a processor for each processor supported by the ingested data buffer.
22. The method of claim 1 , wherein the query processing scheme is defined based on query requirements, determined available resources, and an identification of the one or more dataset sources.
23. The method of claim 1 , wherein defining the query processing scheme comprises generating directed acyclic graph instructions to execute the query on the plurality of processors.
24. The method of claim 1 , wherein defining the query processing scheme comprises generating directed acyclic graph instructions to execute the query on the plurality of processors, and wherein executing the query comprises communicating the directed acyclic graph instructions to the plurality of processors.
25. The method of claim 1 , wherein executing the query comprises receiving results from the plurality of processors, processing the results, and communicating the processed results to a client device.
26. The method of claim 1 , wherein executing the query comprises receiving results from the plurality of processors, collating the results based on time, and communicating the collated results to a client device.
27. The method of claim 1 , wherein executing the query comprises storing results from the plurality of processors in a data store for later use.
28. A computing system, comprising:
one or more processing devices configured to:
receive a query identifying a set of data to be processed and a manner of processing the set of data;
define a query processing scheme for obtaining and processing the set of data;
dynamically allocate a plurality of processors to multiple layers of processors to execute the query based on defining the query processing scheme, wherein dynamically allocating the plurality of processors to multiple layers of processors to execute the query comprises:
dynamically allocate a first subset of the plurality of processors to a first layer of processors to obtain, from one or more dataset sources, the set of data based on the query identifying the set of data to be processed, and
dynamically allocate a second subset of the plurality of processors to a second layer of processors to process the set of data based on the query identifying the manner of processing the set of data, wherein each of the first layer of processors and the second layer of processors can obtain the set of data and process the set of data; and
execute the query based on the query processing scheme.
29. The system of claim 28 , wherein the first layer of processors comprises an intake layer including the first subset of plurality of processors to obtain the set of data from at least one dataset source of the one or more dataset sources.
30. Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system, cause the computing system to:
receive a query identifying a set of data to be processed and a manner of processing the set of data;
define a query processing scheme for obtaining and processing the set of data;
dynamically allocate a plurality of processors to multiple layers of processors to execute the query based on defining the query processing scheme, wherein dynamically allocating the plurality of processors to multiple layers of processors to execute the query comprises:
dynamically allocate a first subset of the plurality of processors to a first layer of processors to obtain, from one or more dataset sources, the set of data based on the query identifying the set of data to be processed, and
dynamically allocate a second subset of the plurality of processors to a second layer of processors to process the set of data based on the query identifying the manner of processing the set of data, wherein each of the first layer of processors and the second layer of processors can obtain the set of data and process the set of data; and
execute the query based on the query processing scheme.