Distributed query compilation and evaluation system and method
A system and method for distributed query compilation and evaluation involving a first table having a first number of partitions and a second table having a second number of partitions. A query operation, such as a join, is performed for each combination of partitions from the first and second tables. The result of each query operation on each combination of partitions is organized into a third number of partitions. A number of tables equal to the third number are generated with each table including one of the partition results from each of the query operation combinations.
1. A computer implemented method of evaluating a query including one or more query operations, the method comprising:
obtaining a compiled query including a join operation against a first table, the first table having a first set of partitions, the first set including an X number of partitions, the query including a partitioning scheme for intermediate tables that includes an N number of partitions;
obtaining a second set of partitions of a second table, the second set including a Y number of partitions;
performing using a query engine the join operation of one or more partitions of the first set against one or more of the partitions of the second set to create a plurality of combination results, wherein if the join operation is a pairwise join, each of the partitions of the first set are joined against corresponding ones of the partitions of the second set such that each join creates a combination result, the number of combination results in the plurality of combination results for the pairwise join being the same as X and Y, wherein if the join operation is a crosswise join, each of the partitions of the first set is joined against each of the partitions of the second set such that each join creates a combination result, the number of combination results in the plurality of combination results for the crosswise join is X*Y;
partitioning each of the plurality of combination results to produce a result set of result partitions for each of the plurality of combination results, each result set including N partitions; and
reducing corresponding ones of the result partitions across the result sets to an intermediate table to produce a set of N intermediate tables, each intermediate table including data from a result partition from each of the result sets.
2. A method according to claim 1 , wherein Y does not equal N at the time of said obtaining a second set of partitions.
3. A method according to claim 2 , further comprising repartitioning the second set to have N number of partitions prior to said performing the join operation such that Y is the same as N.
4. A method according to claim 1 , wherein Y does not equal X at the time of said obtaining a second set of partitions.
5. A method according to claim 4 , further comprising repartitioning the second set to have X number of partitions prior to said performing the join operation such that Y is the same as X.
6. A machine readable hardware storage medium containing machine executable instructions for implementing a method of evaluating a query including one or more query operations, the instructions comprising:
a set of instructions for obtaining a compiled query including a join operation against a first table, the first table having a first set of partitions, the first set including an X number of partitions, the query including a partitioning scheme for intermediate tables that includes an N number of partitions;
a set of instructions for obtaining a second set of partitions of a second table, the second set including a Y number of partitions;
a set of instructions for performing using a query engine the join operation of one or more partitions of the first set against one or more of the partitions of the second set to create a plurality of combination results, wherein if the join operation is a pairwise join, each of the partitions of the first set are joined against corresponding ones of the partitions of the second set such that each join creates a combination result, the number of combination results in the plurality of combination results for the pairwise join being the same as X and Y, wherein if the join operation is a crosswise join, each of the partitions of the first set is joined against each of the partitions of the second set such that each join creates a combination result, the number of combination results in the plurality of combination results for the crosswise join is X*Y;
a set of instructions for partitioning each of the plurality of combination results to produce a result set of result partitions for each of the plurality of combination results, each result set including N partitions; and
a set of instructions for reducing corresponding ones of the result partitions across the result sets to an intermediate table to produce a set of N intermediate tables, each intermediate table including data from a result partition from each of the result sets.
7. A machine readable hardware storage medium according to claim 6 , wherein Y does not equal N at the time of said obtaining a second set of partitions.
8. A machine readable hardware storage medium according to claim 7 , further comprising a set of instructions for repartitioning the second set of partitions to have N number of partitions prior to said performing the join operation such that Y is the same as N.
9. A machine readable hardware storage medium according to claim 6 , wherein Y does not equal X at the time of said obtaining a second set of partitions.
10. A machine readable hardware storage medium according to claim 9 , further comprising a set of instructions for repartitioning the second set of partitions to have X number of partitions prior to said performing the join operation such that Y is the same as X.