Determining records generated by a processing task of a query
Systems and methods are described for determining a quantity of records generated by a processing task of a query executed in a data intake and query. The system receives a query and identifies a processing task of the query and a quantity of records to be processed according to the query. The system determines the number of records generated by the processing task based on the number of records to be processed and a record generation estimate. The system can allocate compute resources or determine a query execution time for at least a portion of the query based on the determined quantity of records generated.
1. A method, comprising:
receiving a query, wherein the query identifies a set of data and a manner of processing the set of data;
identifying a processing task of the query and a quantity of records to be processed according to the processing task;
determining a quantity of records generated by the processing task based on the quantity of records to be processed and a record generation estimate; and
at least one of:
allocating compute resources for at least a portion of the query based on the determined quantity of records generated by the processing task; or
estimating a processing time for the at least a portion of the query based on the determined quantity of records generated by the processing task.
2. The method of claim 1 , wherein the records to be processed are based on events stored in a data store, each event storing a portion of raw machine data associated with a timestamp.
3. The method of claim 1 , wherein the processing task is an extraction rule.
4. The method of claim 1 , wherein the processing task is a data transform.
5. The method of claim 1 , wherein the processing task is configured for execution by one or more worker nodes.
6. The method of claim 1 , wherein identifying the processing task comprises parsing the query to identify a command.
7. The method of claim 1 , wherein identifying the processing task comprises parsing the query to identify a command that generates more records than received.
8. The method of claim 1 , wherein the records to be processed correspond to records received from one or more indexers of a data intake and query system.
9. The method of claim 1 , wherein the records to be processed correspond to records generated by a preceding processing task.
10. The method of claim 1 , wherein determining the quantity of records generated comprises multiplying the quantity of records to be processed by the record generation estimate.
11. The method of claim 1 , wherein determining the quantity of records generated comprises identifying the record generation estimate from a plurality of record generation estimates based on a time range associated with the query and an identification of the processing task.
12. The method of claim 1 , wherein the record generation estimate is obtained from a lookup table that stores a plurality of record generation estimates, and wherein the lookup table stores a time range, processing task identifier, data sourcetype, and record generation ratio for each record generation estimate of the plurality of record generation estimates.
13. The method of claim 1 , wherein the at least a portion of the query corresponds to the processing task.
14. The method of claim 1 , wherein the at least a portion of the query corresponds to the portion of the query that is configured to be executed by one or more worker nodes of a data intake and query system.
15. The method of claim 1 , further comprising allocating the compute resources for the query based on the determined quantity of records generated by the processing task.
16. The method of claim 1 , further comprising allocating the compute resources for the at least a portion of the query, wherein allocating the compute resources comprises allocating the compute resources based the determined quantity of records generated by the processing task and a priority level assigned to the query.
17. The method of claim 1 , further comprising allocating the compute resources for the at least a portion of the query, wherein allocating the compute resources comprises allocating one or more worker nodes to execute a portion of the query based on the determined quantity of records generated by the processing task.
18. The method of claim 1 , further comprising allocating the compute resources for the at least a portion of the query, wherein allocating the compute resources comprises allocating one or more processors to one or more worker nodes to execute a portion of the query based on the determined quantity of records generated by the processing task.
19. The method of claim 1 , wherein the processing task is a first processing task, the quantity of records to be processed is a first quantity of records to be processed, the quantity of records generated is a first quantity of records generated, and the record generation estimate is a first record generation estimate, the method further comprising:
identifying a second processing task of the query and a second quantity of records to be processed according to the second processing task;
determining a second quantity of records generated by the second processing task based on the second quantity of records to be processed and a second record generation estimate; and
allocating compute resources for the query based on the first quantity of records generated and the second quantity of records generated.
20. The method of claim 1 , wherein the processing task is a first processing task, the quantity of records to be processed is a first quantity of records to be processed, the quantity of records generated is a first quantity of records generated, and the record generation estimate is a first record generation estimate, the method further comprising:
identifying a second processing task of the query and a second quantity of records to be processed according to the second processing task;
determining a second quantity of records generated by the second processing task based on the second quantity of records to be processed and a second record generation estimate; and
allocating compute resources for the query based on a larger of the first quantity of records generated and the second quantity of records generated.
21. The method of claim 1 , wherein the processing task is a first processing task, the quantity of records generated is a first quantity of records generated, and the record generation estimate is a first record generation estimate, the method further comprising:
identifying a second processing task of the query;
determining a second quantity of records generated by the second processing task based on the first quantity of records generated and a second record generation estimate; and
allocating compute resources for the query based on the first quantity of records generated and the second quantity of records generated.
22. The method of claim 1 , further comprising estimating the processing time for the query.
23. The method of claim 1 , further comprising allocating compute resources for the at least a portion of the query and estimating the processing time for the query based on the determined quantity of records generated by the processing task and the compute resources allocated for the at least a portion of the query.
24. A computing system of a data intake and query system, the computing system comprising:
memory; and
one or more processing devices coupled to the memory and configured to:
receive a query, wherein the query identifies a set of data and a manner of processing the set of data;
identify a processing task of the query and a quantity of records to be processed according to the processing task;
determine a quantity of records generated by the processing task based on the quantity of records to be processed and a record generation estimate; and
at least one of:
allocate compute resources for at least a portion of the query based on the determined quantity of records generated by the processing task; or
estimate a processing time for the at least a portion of the query based on the determined quantity of records generated by the processing task.
25. The system of claim 24 , wherein the processing task is a first processing task, the quantity of records to be processed is a first quantity of records to be processed, the quantity of records generated is a first quantity of records generated, and the record generation estimate is a first record generation estimate, and wherein the one or more processing devices are further configured to:
identify a second processing task of the query and a second quantity of records to be processed according to the second processing task;
determine a second quantity of records generated by the second processing task based on the second quantity of records to be processed and a second record generation estimate; and
allocate compute resources for the query based on the first quantity of records generated and the second quantity of records generated.
26. The system of claim 24 , wherein the processing task is a first processing task, the quantity of records to be processed is a first quantity of records to be processed, the quantity of records generated is a first quantity of records generated, and the record generation estimate is a first record generation estimate, and wherein the one or more processing devices are further configured to:
identifying a second processing task of the query and a second quantity of records to be processed according to the second processing task;
determining a second quantity of records generated by the second processing task based on the second quantity of records to be processed and a second record generation estimate; and
allocating compute resources for the query based on a larger of the first quantity of records generated and the second quantity of records generated.
27. The system of claim 24 , wherein the processing task is a first processing task, the quantity of records generated is a first quantity of records generated, and the record generation estimate is a first record generation estimate, and wherein the one or more processing devices are further configured to:
identifying a second processing task of the query;
determining a second quantity of records generated by the second processing task based on the first quantity of records generated and a second record generation estimate; and
allocating compute resources for the query based on the first quantity of records generated and the second quantity of records generated.
28. Non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to:
receive a query, wherein the query identifies a set of data and a manner of processing the set of data;
identify a processing task of the query and a quantity of records to be processed according to the processing task;
determine a quantity of records generated by the processing task based on the quantity of records to be processed and a record generation estimate; and
at least one of:
allocate compute resources for at least a portion of the query based on the determined quantity of records generated by the processing task; or
estimate a processing time for the at least a portion of the query based on the determined quantity of records generated by the processing task.
29. The non-transitory computer-readable media of claim 28 , wherein the computer-executable instructions further cause the computing system to estimate the processing time for the query.
30. The non-transitory computer-readable media of claim 28 , wherein the computer-executable instructions further cause the computing system to allocate compute resources for the at least a portion of the query and estimate the processing time for the query based on the determined quantity of records generated by the processing task and the compute resources allocated for the at least a portion of the query.