Approximate query execution system that bounds query execution based on runtime conditions
Systems and methods are disclosed to implement a bounded group by query system that computes approximate time-sliced statistics for groups of records in a dataset according to a group by query. In embodiments, a single pass scan of the dataset is performed to accumulate exact results for a maximum number of groups in a result grouping structure (RGS) and approximate results for additional groups in an approximate result grouping structure (ARGS). RGSs and ARGSs are accumulated by a set of accumulator nodes and provided to an aggregator node, which combines the received structures to generate exact or approximate statistical results for at least a subset of the groups in the dataset. Advantageously, the disclosed query system is able to produce approximate results for at least some of the groups in a single pass of the dataset using size-bounded data structures, without predetermining the actual number of groups in the dataset.
1. A method comprising:
performing, by one or more hardware processors with associated memory that implement a query execution system:
receiving a query specifying to compute statistics for a plurality of groups of records in a dataset; and
executing the query, comprising:
dynamically determining, based on one or more runtime conditions when the query is received, a maximum number of groups for which to calculate exact group statistics according to the query, wherein only approximate group statistics are calculated for additional groups above the maximum number,
allocating a result grouping structure (RGS) to store the exact group statistics and an approximate result grouping structure (ARGS) to store the approximate group statistics,
in a single pass scan of the dataset:
accumulating the exact group statistics in the RGS for the maximum number of groups from the groups of records, and
accumulating approximate group statistics in the ARGS for one or more additional groups above the maximum number from the groups of records;
outputting a response to the query based at least in part on the RGS and the ARGS, wherein
the response is output via a graphical user interface of the query execution system, and
the graphical user interface includes a three-dimensional graph that indicates, in time slices, the exact group statistics for the maximum number of the groups and the approximate group statistics for the one or more additional groups above the maximum number, and
indicating, on the graphical user interface, a recommendation to zoom in on a particular time slice in the three-dimensional graph, wherein the particular time slice is selected based on a number of approximate results in the particular time slice exceeding a configured threshold.
2. The method of claim 1 , wherein
the ARGS stores (a) a plurality of approximate counts for individual groups using a count-min sketch and (b) a plurality of approximate statistics associated with individual ones of the approximate counts.
3. The method of claim 2 , wherein
the ARGS stores approximate statistics for an unbounded number of groups.
4. The method of claim 1 , further comprising the query execution system:
dividing the query into a plurality of accumulator tasks that read individual portions of the dataset in parallel.
5. The method of claim 4 , wherein
the one or more runtime conditions includes a number of portions of the dataset.
6. The method of claim 1 , wherein
the one or more runtime conditions includes a time range specified by the query or a number of time slices specified by the query.
7. The method of claim 1 , wherein
the one or more runtime conditions includes a number and operating conditions of task nodes available to the query execution system to perform query tasks.
8. The method of claim 7 , wherein
the task nodes are individual virtual machine or container instances.
9. The method of claim 1 , wherein
the one or more runtime conditions includes an amount of memory available for executing the query.
10. The method of claim 1 , wherein
the one or more runtime conditions is specified in a configurable policy received by the query execution system.
11. The method of claim 1 , wherein:
the dataset is an event log of events collected from a monitored network; and
the method further comprises performing one or more assessments of the monitored network based on the response to the query to detect one or more conditions of network attack, one or more security vulnerabilities of the monitored network, or one or more compliance violations of the monitored network.
12. A system, comprising:
a query execution system implemented by one or more hardware processors with associated memory, configured to:
receive a query specifying to compute statistics for a plurality of groups of records in a dataset; and
execute the query, including to:
dynamically determine, based on one or more runtime conditions when the query is received, a maximum number of groups for which to calculate exact group statistics according to the query, wherein only approximate group statistics are calculated for additional groups above the maximum number,
allocate a result grouping structure (RGS) to store the exact group statistics and an approximate result grouping structure (ARGS) to store the approximate group statistics,
in a single pass scan of the dataset:
accumulate the exact group statistics in the RGS for the maximum number of groups from the groups of records, and
accumulate approximate group statistics in the ARGS for one or more additional groups above the maximum number from the groups of records;
output a response to the query based at least in part on the RGS and the ARGS, wherein
the response is output via a graphical user interface of the query execution system, and
the graphical user interface includes a three-dimensional graph that indicates, in time slices, the exact group statistics for the maximum number of the groups and the approximate group statistics for the one or more additional groups above the maximum number, and
indicate, on the graphical user interface, a recommendation to zoom in on a particular time slice in the three-dimensional graph, wherein the particular time slice is selected based on a number of approximate results in the particular time slice exceeding a configured threshold.
13. The system of claim 12 , wherein
the ARGS stores (a) a plurality of approximate counts for individual groups using a count-min sketch and (b) a plurality of approximate statistics associated with individual ones of the approximate counts.
14. The system of claim 13 , wherein
the ARGS stores approximate statistics for an unbounded number of groups.
15. The system of claim 12 , wherein
the one or more runtime conditions includes a number of portions of the dataset.
16. The system of claim 12 , wherein
the one or more runtime conditions includes a time range specified by the query or a number of time slices specified by the query.
17. The system of claim 12 , wherein
the one or more runtime conditions includes a number of task nodes available to perform query tasks or an amount of memory available for executing the query.
18. The system of claim 12 , wherein
the one or more runtime conditions is specified in a configurable policy received by the query execution system.