Fast query execution for large datasets
Systems and methods for querying large amounts of data are disclosed. Several different versions of a data feed are provided, ranging from a full set of data to various other versions that are smaller or faster to query (e.g., sampled versions, aggregations, sketches). A machine learning model is trained on features of input queries run against the various versions of the data feed and the corresponding results. The trained model is then applied to a new query to choose, automatically, which version of the data feed to apply the query against. That is, the system can select which version of the data feed to use when executing the given query, optimizing speed and/or compute costs while providing an appropriate level of accuracy for the given query.
1 . A method, comprising:
providing a first version and a second version of a data feed, the second version representing a reduced-size version of the first version;
training a machine learning model on a first set of queries, the training configuring the machine learning model to select between querying against (i) the first version of the data feed or (ii) the second version of the data feed;
where the training is performed using training data including:
one or more features of the first set of queries;
output of a cost function that calculates costs for executing one or more queries in the first set of queries on the first version of the data feed and execution thereof on the second version of the data feed;
after the training:
deploying the trained machine learning model for execution against a second set of queries;
receiving a given query in the second set of queries from a client;
providing one or more features for the given query in the second set of queries to the trained machine learning model;
executing the trained machine learning model against the provided one or more features to produce a selection between the first and second versions of the data feed;
executing the given query in the second set of queries against either the first or second versions of the data feed, as determined by the selection, to obtain a result; and,
providing the result to the client, in response to the given query in the second set of queries.
2 . The method of claim 1 , wherein the second version of the data feed comprises any of: a sampling of the data feed and a data sketch of the data feed.
3 . The method of claim 1 , wherein the cost function is dependent upon one or more of: query execution time, result accuracy relative to a reference version of the data feed, compute cost for query execution, resource consumption for query execution.
4 . The method of claim 1 , wherein the data feed comprises data related to network security events.
5 . The method of claim 1 , wherein the training data further comprise any of:
a query key;
a cardinality measure of a query key;
a number of values associated with a query key; and,
a date range.
6 . The method of claim 1 , wherein the training data further comprise any of:
a record of user action taken after receiving results of one or more of the queries in the first set of queries.
7 . The method of claim 1 , further comprising:
applying the result to a second trained machine learning model to determine whether to rerun the given query in the second set of queries against another version of the data feed other than the selected version;
where the determination is based on a statistic of the result.
8 . The method of claim 7 , wherein the statistic comprises any of:
a number of records in the result, and,
a distribution of the result, and
a comparison of the result to past results for similar or identical queries.
9 . The method of claim 1 , wherein the second version of the data feed is a smaller size version of the first version of the data feed.
10 . A system, comprising:
one or more computers collectively having circuitry forming one or more processors and memory holding computer program instructions for execution on the one or more processors to cause the one or more computers to:
provide a first version and a second version of a data feed, the second version representing a reduced-size version of the first version;
train a machine learning model on a first set of queries, the training configuring the machine learning model to select between querying against (i) the first version of the data feed or (ii) the second version of the data feed;
where the training is performed using training data including:
one or more features of the first set of queries;
output of a cost function that calculates costs for executing one of more queries in the first set of queries on the first version of the data feed and execution thereof on the second version of the data feed;
after the training:
deploy the trained machine learning model for execution against a second set of queries;
receive a given query in the second set of queries from a client;
provide one or more features for the given query in the second set of queries to the machine learning model;
execute the machine learning model against the provided one or more features to produce a selection between the first and second versions of the data feed;
execute the given query in the second set of queries against either the first or second versions of the data feed, as determined by the selection, to obtain a result; and,
provide the result to the client, in response to the given query in the second set of queries.
11 . The system of claim 10 , wherein the second version of the data feed comprises any of: a sampling of the data feed and a data sketch of the data feed.
12 . The system of claim 10 , wherein the cost function is dependent upon one or more of: query execution time, result accuracy relative to a reference version of the data feed, compute cost for query execution, resource consumption for query execution.
13 . The system of claim 10 , wherein the data feed comprises data related to network security events.
14 . The system of claim 10 , wherein the training data further comprise any of:
a query key;
a cardinality measure of a query key;
a number of values associated with a query key; and,
a date range.
15 . The system of claim 10 , wherein the training data further comprise any of:
a record of user action taken after receiving results of one or more queries of the first set of queries.
16 . The system of claim 10 , the execution of the computer program instructions on the one or more processors further causing the system to:
apply the result to a trained second machine learning model to determine whether to rerun the given query in the second set of queries against another version of the data feed other than the selected version;
where the determination is based on a statistic of the result.
17 . The system of claim 16 , wherein the statistic comprises any of:
a number of records in the result, and,
a distribution of the result, and
a comparison of the result to past results for similar or identical queries.
18 . A non-transitory computer readable medium holding computer program instructions executable on one or more processors, the computer program instructions comprising instructions to:
provide a first version and a second version of a data feed, the second version representing a reduced-size version of the first version;
train a machine learning model on a first set of queries, the training configuring the machine learning model to select between querying against (i) the first version of the data feed or (ii) the second version of the data feed;
where the training is performed using training data including:
one or more features of the first set of queries;
output of a cost function that calculates costs for executing one or more queries in the first set of queries on the first version of the data feed and execution thereof on the second version of the data feed;
after the training:
deploy the trained machine learning model for execution against a second set of queries;
receive a given query in the second set of queries from a client;
provide one or more features for the given query in the second set of queries to the machine learning model;
execute the machine learning model against the provided one or more features to produce a selection between the first and second versions of the data feed;
execute the given query in the second set of queries against either the first or second versions of the data feed, as determined by the selection, to obtain a result; and,
provide the result to the client, in response to the given query in the second set of queries.