Bucket data distribution for exporting data to worker nodes
Systems and methods are described for exporting bucket data from one or more buckets to one or more worker nodes. The system can identify data from different bucket data from buckets stored in a data intake and query system that is to be processed by one or more worker nodes. The system can allocate one or more execution resources, such as a processing pipeline, to process and export the bucket data from the buckets. The system can assign bucket data corresponding to individual buckets to the execution resource based on a bucket distribution policy. The indexer can export the bucket data to the worker nodes for further processing based on the bucket data-execution resource assignment.
1. A method, comprising:
receiving a query, wherein the query identifies a set of data and a manner of processing the set of data;
identifying one or more buckets associated with the query;
identifying one or more bucket data associated with the query, wherein bucket data is identified for each bucket of the one or more buckets;
determining one or more execution resources to allocate to process the one or more bucket data;
assigning each of the one or more bucket data to an execution resource of the one or more execution resources based on content of the one or more bucket data; and
exporting the one or more bucket data to one or more worker nodes based on the assigning, wherein the one or more worker nodes process the one or more bucket data and provide query results based on the query.
2. The method of claim 1 , wherein the query is received at an indexer of a data intake and query system.
3. The method of claim 1 , wherein the query is a subquery of a query received by a data intake and query system.
4. The method of claim 1 , wherein the set of data is a subset of data of a data intake and query system.
5. The method of claim 1 , wherein the one or more buckets correspond to one or more file system directories.
6. The method of claim 1 , wherein exporting comprises processing the one or more bucket data based on the query to provide one or more processed bucket data, and exporting the one or more processed bucket data based on the assigning.
7. The method of claim 1 , wherein the one or more buckets associated with the query are identified based on one or more query parameters of the query.
8. The method of claim 1 , wherein the one or more buckets associated with the query are identified based on at least one of a partition or a time range identified by the query.
9. The method of claim 1 , wherein the one or more bucket data associated with the query are identified based on one or more query parameters of the query.
10. The method of claim 1 , wherein the one or more bucket data associated with the query are identified based on at least one of a partition, a time range, a field, a field-value pair, or a keyword, identified by the query.
11. The method of claim 1 , wherein particular bucket data of the one or more bucket data is identified based on a comparison of one or more query parameters of the query with data of an inverted index associated with a particular bucket of the one or more buckets.
12. The method of claim 1 , wherein identifying the one or more bucket data comprises identifying a quantity of events associated with the query for each bucket of the one or more buckets.
13. The method of claim 1 , wherein determining one or more execution resources comprises determining the one or more execution resources based on an execution resource allocation policy.
14. The method of claim 1 , wherein determining one or more execution resources comprises allocating the one or more execution resources based on a lesser of a quantity of the one or more buckets, a quantity of available execution resources, and a threshold quantity.
15. The method of claim 1 , wherein the one or more execution resources comprise one or more processors.
16. The method of claim 1 , wherein the one or more execution resources comprise one or more processing pipelines.
17. The method of claim 1 , wherein the assigning comprises assigning each of the one or more bucket data to an execution resource of the one or more execution resources based on a quantity of events of each of the one or more bucket data.
18. The method of claim 1 , wherein the assigning comprises assigning each of the one or more bucket data to an execution resource of the one or more execution resources based on a quantity of events of each of the one or more bucket data to reduce a difference between a largest quantity of events assigned to a first execution resource and a smallest quantity of events assigned to a second execution resource.
19. The method of claim 1 , wherein the assigning comprises assigning each of the one or more bucket data to an execution resource of the one or more execution resources based on a quantity of events of each of the one or more bucket data to approximate an equal distribution of events to the one or more execution resources.
20. A computing system of a data intake and query system, the computing system comprising:
memory; and
one or more processing devices coupled to the memory and configured to:
receive a query, wherein the query identifies a set of data and a manner of processing the set of data;
identify one or more buckets associated with the query;
identify one or more bucket data associated with the query, wherein bucket data is identified for each bucket of the one or more buckets;
determine one or more execution resources to allocate to process the one or more bucket data;
assign each of the one or more bucket data to an execution resource of the one or more execution resources based on content of the one or more bucket data; and
export the one or more bucket data to one or more worker nodes based on the assigning, wherein the one or more worker nodes process the one or more bucket data and provide query results based on the query.
21. The system of claim 20 , wherein the one or more buckets associated with the query are identified based on one or more query parameters of the query.
22. The system of claim 20 , wherein the one or more buckets associated with the query are identified based on at least one of a partition or a time range identified by the query.
23. The system of claim 20 , wherein particular bucket data of the one or more bucket data is identified based on a comparison of one or more query parameters of the query with data of an inverted index associated with a particular bucket of the one or more buckets.
24. The system of claim 20 , wherein to assign each of the one or more bucket data to an execution resource the one or more processing devices are configured to assign each of the one or more bucket data to an execution resource of the one or more execution resources based on a quantity of events of each of the one or more bucket data.
25. The system of claim 20 , wherein to assign each of the one or more bucket data to an execution resource the one or more processing devices are configured to assign each of the one or more bucket data to an execution resource of the one or more execution resources based on a quantity of events of each of the one or more bucket data to reduce a difference between a largest quantity of events assigned to a first execution resource and a smallest quantity of events assigned to a second execution resource.
26. The system of claim 20 , wherein to assign each of the one or more bucket data to an execution resource the one or more processing devices are configured to assign each of the one or more bucket data to an execution resource of the one or more execution resources based on a quantity of events of each of the one or more bucket data to approximate an equal distribution of events to the one or more execution resources.
27. Non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to:
receive a query, wherein the query identifies a set of data and a manner of processing the set of data;
identify one or more buckets associated with the query;
identify one or more bucket data associated with the query, wherein bucket data is identified for each bucket of the one or more buckets;
determine one or more execution resources to allocate to process the one or more bucket data;
assign each of the one or more bucket data to an execution resource of the one or more execution resources based on content of the one or more bucket data; and
export the one or more bucket data to one or more worker nodes based on the assigning, wherein the one or more worker nodes process the one or more bucket data and provide query results based on the query.
28. The non-transitory computer-readable media of claim 27 , wherein to determine one or more execution resources the computer-executable instructions cause the computing system to allocate the one or more execution resources based on a lesser of a quantity of the one or more buckets, a quantity of available execution resources, and a threshold quantity.
29. The non-transitory computer-readable media of claim 27 , wherein to determine one or more execution resources the computer-executable instructions cause the computing system to determine the one or more execution resources based on an execution resource allocation policy.
30. The non-transitory computer-readable media of claim 27 , wherein the one or more execution resources comprise one or more processing pipelines.