System and method for optimizing large database management systems using bloom filter
A large highly parallel database management system includes thousands of nodes storing huge volume of data. The database management system includes a query optimizer for optimizing data queries. The optimizer estimates the column cardinality of a set of rows based on estimated column cardinalities of disjoint subsets of the set of rows. For a particular column, the actual column cardinality of the set of rows is the sum of the actual column cardinalities of the two subsets of rows. The optimizer creates two respective Bloom filters from the two subsets, and then combines them to create a combined Bloom filter using logical OR operations. The actual column cardinality of the set of rows is estimated using a computation from the combined Bloom filter.
1. A method of optimizing data queries for execution by a computer of a database management system, the method comprising:
estimating a first column cardinality of a first column based on a first subset of rows of a set of rows of the first column to produce a first estimated first column cardinality;
estimating a second first column cardinality of the first column based on a second subset of rows of the set of rows to produce a second estimated first column cardinality, wherein the first subset of rows and the second subset of rows are disjoint subsets of the set of rows, and wherein a column cardinality of the first column is a number of distinct values within the set of rows of the first column;
generating a first Bloom filter based on the first subset of rows;
generating a second Bloom filter based on the second subset of rows;
combining the first Bloom filter and the second Bloom filter to create a combined Bloom filter; and
determining an estimated column cardinality of the first column based on the combined Bloom filter.
2. The method of claim 1 further comprising:
logically ORing corresponding bits of the first Bloom filter and the second Bloom filter to produce the combined Bloom filter.
3. The method of claim 1 , wherein the generating the first Bloom filter is further based on one or more of the set of rows, the first estimated first column cardinality, and a frequency of frequencies for each distinct value within the first subset of rows.
4. The method of claim 1 further comprising:
randomly sampling rows of the set of rows to produce the first subset of rows; and
randomly sampling second rows of the set of rows to produce the second subset of rows.
5. The method of claim 1 further comprising:
determining a cost estimate for an operation of a data query of the data queries based on the estimated column cardinality.