Data conditioning for dataset destination
Systems and methods are disclosed for processing queries against one or more dataset sources utilizing dynamically allocated partitions operating on one or more worker nodes. The results of the processing are stored in a dataset destination. The queries can identify data in the one or more dataset sources for processing and a manner for processing the data. In addition, the queries can identify the dataset destination for storing results of the query. To process the query, a query coordinator can dynamically allocate partitions operating on worker nodes to retrieve data for processing, process the data, and communicate the data to the dataset sources. In addition, the query coordinator can dynamically allocate partitions based on an identification of the dataset destination.
1. A method comprising:
receiving, at a data intake and query system, a query identifying a set of data to be processed, a manner of processing the set of data, and a dataset destination for storing results of processing the set of data;
determining a first number of processors associated with the dataset destination;
defining, by the data intake and query system, a query processing scheme for obtaining and processing the set of data and storing the results of processing the set of data, the query processing scheme including a dynamic allocation of multiple layers of processors to execute various phases of the query, wherein each layer of the multiple layers of processors is dynamically allocated a particular number of processors of a plurality of processors to execute a respective phase of the various phases of the query, wherein the multiple layers of processors comprise a storage layer of processors to prepare and communicate the results of processing the set of data for storage in the dataset destination, wherein the storage layer of processors is dynamically allocated a second number of processors of the plurality of processors based on the determined first number of processors associated with the dataset destination, wherein each of the second number of processors dynamically allocated to the storage layer of processors are dynamically allocable to at least a second processor of the processors associated with the dataset destination, wherein a first processor of the second number of processors dynamically allocated to the storage layer of processors is dynamically allocated to communicate at least partial results of processing the set of data to the at least the second processor based on dynamically allocating the second number of processors to the storage layer of processors, wherein the storage layer of processors communicate the results of processing the set of data to the processors associated with the dataset destination; and
executing the query based on the query processing scheme.
2. The method of claim 1 , wherein the dataset destination is a first dataset destination and the storage layer is a first storage layer, wherein the multiple layers of processors further comprise a second storage layer of processors to prepare and communicate the results of processing the set of data to a second dataset destination, wherein the second storage layer of processors is dynamically allocated a third number of processors of the plurality of processors.
3. The method of claim 1 , wherein defining the query processing scheme comprises determining a processing capability of the dataset destination, and wherein the dynamic allocation of the multiple layers of processors is based on the determining the processing capability.
4. The method of claim 1 , wherein the dynamic allocation of the multiple layers of processors is based on the determining the first number of processors associated with the dataset destination.
5. The method of claim 1 , wherein the dataset destination comprises an external data source.
6. The method of claim 1 , wherein the dataset destination comprises an external data source and the storage layer of processors includes the second number of processors to communicate the results of processing the set of data to the external data source.
7. The method of claim 1 , wherein the dataset destination comprises an external data source and the storage layer of processors includes a processor for each processor associated with the external data source.
8. The method of claim 1 , wherein the dataset destination comprises an ingested data buffer.
9. The method of claim 1 , wherein the dataset destination comprises an ingested data buffer and the storage layer of processors includes the second number of processors to communicate the results of processing the set of data to the ingested data buffer.
10. The method of claim 1 , wherein the dataset destination comprises an ingested data buffer and the storage layer of processors includes a processor for each processor associated with the ingested data buffer.
11. The method of claim 1 , wherein the dataset destination comprises a query acceleration data store.
12. The method of claim 1 , wherein the dataset destination comprises a query acceleration data store and the storage layer of processors includes the second number of processors to communicate the results of processing the set of data to the query acceleration data store.
13. The method of claim 1 , wherein the query processing scheme is defined based on query requirements, determined available resources, and an identification of the dataset destination.
14. The method of claim 1 , wherein defining the query processing scheme comprises generating instructions for execution by the multiple layers of processors, and wherein executing the query comprises communicating the instructions to the multiple layers of processors.
15. The method of claim 1 , wherein defining the query processing scheme comprises generating directed acyclic graph instructions for the multiple layers of processors.
16. The method of claim 1 , wherein defining the query processing scheme comprises generating directed acyclic graph instructions for the multiple layers of processors, and wherein executing the query comprises communicating the directed acyclic graph instructions to the multiple layers of processors.
17. The method of claim 1 , wherein defining the query processing scheme comprises monitoring the dataset destination for activity and accessibility.
18. The method of claim 1 , wherein executing the query comprises monitoring the dataset destination.
19. The method of claim 1 , wherein executing the query comprises monitoring the dataset destination and allocating an additional processor to the storage layer of processors based on a determination that an additional processor associated with the dataset destination is available.
20. The method of claim 1 , wherein executing the query comprises monitoring the dataset destination and deallocating a processor from the storage layer of processors based on a determination that a processor associated with the dataset destination is not available.
21. The method of claim 1 , wherein defining the query processing scheme comprises determining that the dataset destination is associated with a first single processor, wherein the second number of processors comprises a second single processor based on the first single processor associated with the dataset destination.
22. The method of claim 1 , wherein the dataset destination comprises an external data source, wherein defining the query processing scheme comprises: determining that the external data source is associated with a single processor; wherein the second number of processors comprises at least two processors, wherein each processor of the second number of processors is configured to concurrently communicate a subset of the results of processing the set of data to the single processor associated with the external data source.
23. The method of claim 1 , wherein defining the query processing scheme comprises determining that the dataset destination is associated with a single processor; wherein the second number of processors comprises at least two processors, wherein each processor of the second number of processors is configured to concurrently communicate a subset of the results of processing the set of data to the single processor associated with the dataset destination.
24. The method of claim 1 , wherein the multiple layers of processors further comprise an intake layer of processors including a third number of the plurality of processors to receive data from at least one dataset source.
25. The method of claim 1 , wherein the multiple layers of processors further comprise an intake layer of processors including a third number of the plurality of processors to receive data from a dataset source, a processing layer of processors including a fourth number of the plurality of processors to receive and process data from the intake layer of processors, and a collector layer of processors including a fifth number of the plurality of processors to collect data from the processing layer of processors, wherein the storage layer of processors receives data from the collector layer of processors.
26. The method of claim 1 , wherein the multiple layers of processors further comprise a branch layer of processors including at least one processor to communicate data to multiple locations, wherein the storage layer of processors receives data from the branch layer of processors.
27. A computing system, comprising:
one or more processing devices configured to:
receive a query identifying a set of data to be processed, a manner of processing the set of data, and a dataset destination for storing results of processing the set of data;
determine a first number of processors associated with the dataset destination;
define a query processing scheme for obtaining and processing the set of data and storing the results of processing the set of data, the query processing scheme including a dynamic allocation of multiple layers of processors to execute various phases of the query, wherein each layer of the multiple layers of processors is dynamically allocated a particular number of processors of a plurality of processors to execute a respective phase of the various phases of the query, wherein the multiple layers of processors comprise a storage layer of processors to prepare and communicate the results of processing the set of data for storage in the dataset destination, wherein the storage layer of processors is dynamically allocated a second number of processors of the plurality of processors based on the determined first number of processors associated with the dataset destination, wherein each of the second number of processors dynamically allocated to the storage layer of processors are dynamically allocable to at least second processor of the processors associated with the dataset destination, wherein a first processor of the second number of processors dynamically allocated to the storage layer of processors is dynamically allocated to communicate at least partial results of processing the set of data to the at least ft the second processor based on dynamically allocating the second number of processors to the storage layer of processors, wherein the storage layer of processors communicate the results of processing the set of data to the processors associated with the dataset destination; and
execute the query based on the query processing scheme.
28. The system of claim 27 , wherein the multiple layers of processors further comprise a branch layer of processors including at least one processor to communicate data to multiple locations, wherein the storage layer of processors receives data from the branch layer of processors.
29. The system of claim 27 , wherein the dynamic allocation of the multiple layers of processors is based on the first number of processors associated with the dataset destination.
30. Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system, cause the computing system to:
receive a query identifying a set of data located to be processed, a manner of processing the set of data, and a dataset destination for storing results of processing the set of data;
determine a first number of processors associated with the dataset destination;
define a query processing scheme for obtaining and processing the set of data and storing the results of processing the set of data, the query processing scheme including a dynamic allocation of multiple layers of processors to execute various phases of the query, wherein each layer of the multiple layers of processors is dynamically allocated a particular number of processors of a plurality of processors to execute a respective phase of the various phases of the query, wherein the multiple layers of processors comprise a storage layer of processors to prepare and communicate the results of processing the set of data for storage in the dataset destination, wherein the storage layer of processors is dynamically allocated a second number of processors of the plurality of processors based on the determined first number of processors associated with the dataset destination, wherein each of the second number of processors dynamically allocated to the storage layer of processors are dynamically allocable to at least a second processor of the processors associated with the dataset destination, wherein a first processor of the second number of processors dynamically allocated to the storage layer of processors is dynamically allocated to communicate at least partial results of processing the set of data to the at least the second processor based on dynamically allocating the second number of processors to the storage layer of processors, wherein the storage layer of processors communicate the results of processing the set of data to the processors associated with the dataset destination; and
execute the query based on the query processing scheme.