Adaptive distribution method for hash operations
A method, apparatus, and system for join operations of a plurality of relations that are distributed over a plurality of storage locations over a network of computing components.
1. A method of performing an implementation of a join operation, comprising:
separating a join operation into a build operation and a probe operation;
determining, by a processing device, a distribution method to distribute at least one of a first relation and a second relation to the probe operation based at least in part on an actual size of the second relation; and
performing the probe operation on the first relation and the second relation.
2. The method of claim 1 , wherein determining the distribution method to distribute at least one of the first relation and the second relation is based on a cost metric associated with performing the distribution method on the first relation and the second relation.
3. The method of claim 2 , wherein the cost metric is a computational cost or a memory cost associated with the distribution method and the actual size of the second relation.
4. The method of claim 2 , wherein determining the distribution method to distribute at least one of the first relation and the second relation based on the cost metric comprises:
determining that a broadcast join is a most efficient type of join operation based on the cost metric.
5. The method of claim 4 , further comprising distributing the second relation using the broadcast join, wherein using the broadcast join comprises assigning at least part of the second relation to two or more processing nodes associated with the first relation.
6. The method of claim 5 , wherein distributing the second relation comprises:
duplicating the second relation across the two or more processing nodes associated with the first relation; and
forwarding a local partition of the first relation to a local instance of a probe operator of the probe operation for performing a join query.
7. The method of claim 1 , further comprising:
performing the build operation on the second relation, wherein performing the build operation comprises building a hash table for the second relation.
8. The method of claim 7 , wherein the actual size of the second relation is determined prior to performing the probe operation.
9. The method of claim 7 , wherein performing the probe operation on the first relation and the second relation comprises:
reading an inner stream of the first relation and the second relation; and
probing the hash table generated by the build operation to identify tuples of the first relation and the second relation that satisfy a join condition.
10. The method of claim 1 , wherein the join operation is an equijoin operation.
11. A system comprising:
a memory to store a plurality of relations; and
a processor, operatively coupled to the memory, the processor to:
separate a join operation into a build operation and a probe operation;
determine a distribution method to distribute at least one of a first relation and a second relation to the probe operation based at least in part on an actual size of the second relation; and
performing the probe operation on the first relation and the second relation.
12. The system of claim 11 , wherein the processor is to determine the distribution method to distribute at least one of the first relation and the second relation based on a cost metric associated with performing the distribution method on the first relation and the second relation.
13. The system of claim 12 , wherein the cost metric is a computational cost or a memory cost associated with the distribution method and the actual size of the second relation.
14. The system of claim 12 , wherein to determine the distribution method to distribute at least one of the first relation and the second relation based on the cost metric, the processor is to:
determine that a broadcast join is a most efficient type of join operation based on the cost metric.
15. The system of claim 14 , wherein the processor is further to distribute the second relation using the broadcast join, wherein to use the broadcast join the processor is to assign at least part of the second relation to two or more processing nodes associated with the first relation.
16. The system of claim 15 , wherein to distribute the second relation the processor is to:
duplicate the second relation across the two or more processing nodes associated with the first relation; and
forward a local partition of the first relation to a local instance of a probe operator for performing a join query.
17. The system of claim 11 , wherein the processor is further to:
perform the build operation on the second relation, wherein performing the build operation comprises building a hash table for the second relation.
18. The system of claim 17 , wherein the actual size of the second relation is determined prior to performing the probe operation.
19. The system of claim 17 , wherein to perform the probe operation on the first relation and the second relation comprises, the processor is to:
read an inner stream of the first relation and the second relation; and
probe the hash table generated by the build operation to identify tuples of the first relation and the second relation that satisfy a join condition.
20. The system of claim 11 , wherein the join operation is an equijoin operation.
21. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processor, cause the processor to:
separate a join operation into a build operation and a probe operation;
determine, by a processor, a distribution method to distribute at least one of a first and a second relation to the probe operation based at least in part on an actual size of the second relation; and
perform the probe operation on the first relation and the second relation.
22. The non-transitory computer readable medium of claim 21 , wherein to determine the distribution method to distribute at least one of the first relation and the second relation is further based on a cost metric associated with the distribution method of the first relation and the second relation.
23. The non-transitory computer readable medium of claim 22 , wherein the cost metric is a computational cost or a memory cost associated with the distribution method and the actual size of the second relation.
24. The non-transitory computer readable medium of claim 22 , wherein to determine the distribution method to distribute at least one of the first relation and second relation based on the cost metric the processor is to:
determine that a broadcast join is a most efficient type of join operation based on the cost metric.
25. The non-transitory computer readable medium of claim 24 , wherein the processor is further to distribute the second relation using the broadcast join, wherein to use the broadcast join the processor is to assign at least part of the second relation to two or more processing nodes associated with the first relation.
26. The non-transitory computer readable medium of claim 25 , wherein to distribute the second relation the processor is to:
duplicate the second relation across the two or more processing nodes associated with the first relation; and
forward a local partition of the first relation to a local instance of a probe operator for performing a join query.
27. The non-transitory computer readable medium of claim 21 , wherein the processor is further to:
perform the build operation on the second relation, wherein performing the build operation comprises building a hash table for the second relation.
28. The non-transitory computer readable medium of claim 27 , wherein the actual size of the second relation is determined prior to performing the probe operation.
29. The non-transitory computer readable medium of claim 27 , wherein to perform the probe operation on the first relation and the second relation comprises, the processor is to:
read an inner stream of the first relation and the second relation; and
probe the hash table generated by the build operation to identify tuples of the first relation and the second relation that satisfy a join condition.
30. The non-transitory computer readable medium of claim 21 , wherein the join operation is an equijoin operation.