Query scheduling based on a query-resource allocation and resource availability
Systems and methods are described for scheduling a query for execution. The system receives and parses a query to identify one or more portions of the query. The system determines a resource allocation for each portion of the query, and determines an availability of compute resources for the different portions of the query. Based on the resource allocation and the availability of compute resources, the system schedules the query.
1. A method, comprising:
receiving a query, wherein the query identifies a set of data and a manner of processing the set of data;
determining a query-resource allocation for one or more portions of the query;
determining compute resource availability for the one or more portions of the query; and
scheduling the query based on the query-resource allocation and the determined compute resource availability.
2. The method of claim 1 , wherein determining a query-resource allocation comprises determining a plurality of query-resource allocations for a plurality of portions of the query.
3. The method of claim 1 , wherein determining a query-resource allocation comprises identifying a portion of the query and determining the query-resource allocation for the portion of the query.
4. The method of claim 1 , wherein determining a first query-resource allocation that includes an allocation of a first quantity of compute resources and determining a second query-resource allocation that includes an allocation of a second quantity of compute resources that is less than the first query-resource allocation, the method further comprising:
determining that there are insufficient compute resources to execute the query based on the first quantity of compute resources; and
scheduling the query for execution based on the second query-resource allocation.
5. The method of claim 1 , wherein determining a query-resource allocation comprises identifying a portion of the query and determining the query-resource allocation for the portion of the query based on an identification of components of a data intake and query system that are to execute the portion of the query.
6. The method of claim 1 , wherein determining a query-resource allocation comprises identifying a portion of the query and determining the query-resource allocation for the portion of the query based on a query type of the query.
7. The method of claim 1 , wherein determining a query-resource allocation comprises identifying a portion of the query and determining the query-resource allocation for the portion of the query based on at least one of a quantity of indexers that store at least a portion of the set of data, a prioritization level, or a quantity worker nodes to receive results of the portion of the query.
8. The method of claim 1 , wherein determining a query-resource allocation comprises identifying a portion of the query and determining the query-resource allocation for the portion of the query based on at least one of a quantity of indexers that store at least a portion of the set of data or a weighting factor.
9. The method of claim 1 , wherein determining a query-resource allocation comprises identifying a portion of the query and determining the query-resource allocation for the portion of the query based on at least one of a quantity of records to be processed as a result of the portion of the query, a prioritization level of the query, or a query allocation threshold.
10. The method of claim 1 , wherein determining a query-resource allocation comprises identifying a portion of the query and determining the query-resource allocation for the portion of the query based on one or more processing tasks of the query.
11. The method of claim 1 , wherein determining a query-resource allocation comprises identifying a portion of the query and determining the query-resource allocation for the portion of the query based a quantity of records to be processed as a result of a processing task of the query.
12. The method of claim 1 , wherein determining a query-resource allocation comprises identifying a portion of the query and allocating a first group of compute resources for a first processing task of the portion of the query and allocating a second group of compute resources for a second processing task of the portion of the query.
13. The method of claim 1 , wherein determining a query-resource allocation comprises:
identifying a portion of the query,
determining a quantity of records to be processed according to a processing task of the portion of the query;
assigning a size category of a plurality of size categories to the processing task based on the quantity of records to be processed according to the processing task; and
determining a query-resource allocation for the processing task based on the size category assigned to the processing task.
14. The method of claim 1 , wherein determining the compute resource availability comprises determining a plurality of compute resource availability for a plurality of portions of the query.
15. The method of claim 1 , wherein determining the compute resource availability comprises identifying a portion of the query and determining the compute resource availability for the portion of the query.
16. The method of claim 1 , wherein determining the compute resource availability comprises identifying a portion of the query and determining the compute resource availability for the portion of the query based on a quantity of compute resources of a plurality of indexers and a quantity of the compute resources allocated to one or more other queries.
17. A computing system of a data intake and query system, the computing system comprising:
memory; and
one or more processing devices coupled to the memory and configured to:
receive a query, wherein the query identifies a set of data and a manner of processing the set of data;
determine a query-resource allocation for one or more portions of the query;
determine compute resource availability for the one or more portions of the query; and
schedule the query based on the query-resource allocation and the determined compute resource availability.
18. The system of claim 17 , wherein to determine a query-resource allocation, the one or more processing devices are configured to determine a plurality of query-resource allocations for a plurality of portions of the query.
19. The system of claim 17 , wherein to determine a query-resource allocation, the one or more processing devices are configured to determine a first query-resource allocation that includes an allocation of a first quantity of compute resources and determining a second query-resource allocation that includes an allocation of a second quantity of compute resources that is less than the first query-resource allocation, and wherein the one or more processing devices are further configured to:
determine that there are insufficient compute resources to execute the query based on the first quantity of compute resources; and
schedule the query for execution based on the second query-resource allocation.
20. The system of claim 17 , wherein to determine a query-resource allocation, the one or more processing devices are configured to identify a portion of the query and determine the query-resource allocation for the portion of the query based on an identification of components of a data intake and query system that are to execute the portion of the query.
21. The system of claim 17 , wherein to determine a query-resource allocation, the one or more processing devices are configured to identify a portion of the query and determine the query-resource allocation for the portion of the query based on at least one of a quantity of indexers that store at least a portion of the set of data, a prioritization level, or a quantity worker nodes to receive results of the portion of the query.
22. The system of claim 17 , wherein to determine a query-resource allocation, the one or more processing devices are configured to identify a portion of the query and determine the query-resource allocation for the portion of the query based on at least one of a quantity of records to be processed as a result of the portion of the query, a prioritization level of the query, or a query allocation threshold.
23. Non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to:
receive a query, wherein the query identifies a set of data and a manner of processing the set of data;
determine a query-resource allocation for one or more portions of the query;
determine compute resource availability for the one or more portions of the query; and
schedule the query based on the query-resource allocation and the determined compute resource availability.
24. The non-transitory computer-readable media of claim 23 , wherein to determine a query-resource allocation, the computer-executable instructions cause the computing system to identify a portion of the query and determine the query-resource allocation for the portion of the query based on one or more processing tasks of the query.
25. The non-transitory computer-readable media of claim 23 , wherein to determine a query-resource allocation, the computer-executable instructions cause the computing system to identify a portion of the query and determine the query-resource allocation for the portion of the query based a quantity of records to be processed as a result of a processing task of the query.
26. The non-transitory computer-readable media of claim 23 , wherein to determine a query-resource allocation, the computer-executable instructions cause the computing system to identify a portion of the query and allocating a first group of compute resources for a first processing task of the portion of the query and allocate a second group of compute resources for a second processing task of the portion of the query.
27. The non-transitory computer-readable media of claim 23 , wherein to determine a query-resource allocation, the computer-executable instructions cause the computing system to:
identify a portion of the query,
determine a quantity of records to be processed according to a processing task of the portion of the query;
assign a size category of a plurality of size categories to the processing task based on the quantity of records to be processed according to the processing task; and
determine a query-resource allocation for the processing task based on the size category assigned to the processing task.
28. The non-transitory computer-readable media of claim 23 , wherein to determine the compute resource availability, the computer-executable instructions cause the computing system to determine a plurality of compute resource availability for a plurality of portions of the query.
29. The non-transitory computer-readable media of claim 23 , wherein to determine the compute resource availability, the computer-executable instructions cause the computing system to identifying a portion of the query and determining the compute resource availability for the portion of the query.
30. The non-transitory computer-readable media of claim 23 , wherein to determine the compute resource availability, the computer-executable instructions cause the computing system to identify a portion of the query and determining the compute resource availability for the portion of the query based on a quantity of compute resources of a plurality of indexers and a quantity of the compute resources allocated to one or more other queries.