Managing complex queries with predicates
A first query is detected. The first query is for a storage device, and pulls a result set from the storage device using a set of predicates. The first query satisfies a size criterion. A new set of predicates of the result set is identified. The new set of predicates does not include any predicates of the original set of predicates. A second query which is the same as the first query is detected. The new set of predicates is used to determine the result set.
1. A method of processing complex queries, comprising:
detecting a first query from a first client device directed towards a dataset, wherein the first query includes an initial set of predicates and results in a first result set, wherein an initial set of tuples are initially gathered from the dataset in response to the first query, wherein each tuple of the initial set of tuples satisfies at least one of the predicates of the initial set of predicates, wherein the initial set of tuples is reduced to the first result set as predicates of the initial set of predicates eliminate tuples;
storing a query plan and query statistics of the first query in the database;
determining that the initial set of tuples includes an amount of tuples that satisfies a size criterion;
in response to determining that the amount of tuples satisfies the size criterion, identifying a set of columns from the first result set that do not correspond to a predicate of the initial set of predicates;
executing both a set of min/max filters and a set of bloom filters on the first result set, wherein each bloom filter of the set of bloom filters is derived from a respective column of the set of columns and each min/max filter of the set of min/max filters is derived from a respective column of the set of columns;
determining that a subset of the set of bloom filters would filter a threshold percentage of tuples from the initial set of tuples;
determining that a subset of the min/max filters would filter the threshold percentage of tuples from the initial set of tuples;
identifying both the subset of the set of bloom filters and the subset of the min/max filters as new predicates;
detecting a second query from a second client device directed towards the dataset, wherein the second query is the same as the first query;
evaluating the new predicates immediately before executing the second query;
detecting an indicia of ineffectiveness regarding the new predicates while evaluating the new predicates;
in response to detecting the indicia of ineffectiveness, disabling the new predicates at runtime for the second query, wherein disabling the new predicates results in the new predicates not being used in the second query;
testing the new predicates following the conclusion of the runtime for the second query;
verifying that each of the new predicates filter the threshold amount of tuples from the initial set of tuples;
detecting a third query from a third client device directed towards the dataset, wherein the third query is the same as both the first query and the second query; and
utilizing the new predicates, the query plan, and the query statistics to process the third query and determine a third result set for the third query, wherein the new predicates are used in response to verifying that each of the new predicates filter the threshold amount of tuples from the initial set of tuples.