Adaptive distribution for hash operations
A method, apparatus, and system for join operations of a plurality of relations that are distributed over a plurality of storage locations over a network of computing components.
1. A method for joining relations, the method comprising:
receiving a relational join query for a join operation for a plurality of relations, wherein the plurality of relations comprises a first relation and a second relation;
determining, by a processing device, a type of join to be performed on the plurality of relations based on an actual size of the second relation and an estimated size of the first relation, wherein the type of join is a broadcasting join or a re-partitioning join; and
managing, by the processing device, a state of one or more communication links between at least one build operator and at least one probe operator based on the type of join to be performed to perform the join operation.
2. The method of claim 1 , wherein managing the state of the one or more communication links comprises, in response to determining to join the plurality of relations via the broadcasting join, converting a communication link of the one or more communication links between the at least one build operator and the at least one probe operator into a synchronous link.
3. The method of claim 2 , wherein the synchronous link denotes a local one-to-one link between the at least one build operator and the at least one probe operator, and wherein the at least one probe operator performs the broadcasting join.
4. The method of claim 3 , wherein converting the communication link comprises converting the communication link of the one or more communication links between the at least one build operator and the at least one probe operator from an initially inactive and adaptive state to the synchronous link.
5. The method of claim 1 , wherein managing the state of the one or more communication links comprises, in response to determining to join the plurality of relations via the re-partitioning join, converting a communication link of the one or more communication links between the at least one build operator and the at least one probe operator into a partition link.
6. The method of claim 5 , wherein the first relation and the second relation are partitioned among a plurality of processors.
7. The method of claim 5 , further comprising during a re-partition join, sending a partition of the second relation to a local probe operator instance as a single pointer to a block of memory or a file on a disk.
8. The method of claim 7 , wherein partitioning the second relation comprises a partitioning operator comprising a hash function over at least one column of the second relation.
9. The method of claim 1 , wherein during a broadcasting join, a local partition of the first relation is forwarded to a local instance of the at least one probe operator for performing the relational join query.
10. The method of claim 9 , wherein the broadcasting join further comprises broadcasting each partition of a plurality of partitions of the second relation to every instance of the at least one probe operator in a distributed system.
11. The method of claim 1 , wherein during a re-partitioning join, a partitioning operation partitions the first relation across a plurality of processing nodes of a distributed system and the re-partitioning join is performed by the at least one probe operator at each of the plurality of processing nodes.
12. A system for joining relations, the system comprising:
a memory to store a plurality of relations; and
a processing device, operatively coupled to the memory, the processing device to:
receive a relational join query for a join operation comprising a predicate and a plurality of relations, wherein the plurality of relations comprises a first relation and a second relation;
determine a type of join to be performed on the plurality of relations based on an actual size of the second relation and an estimated size of the first relation, wherein the type of join is a broadcasting join or a re-partitioning join; and
manage a state of one or more communication links between at least one build operator and at least one probe operator based on the type of join to be performed to perform the join operation.
13. The system of claim 12 , wherein to manage the state of the one or more communication links, the processing device is to, in response to determining to join the plurality of relations via the broadcasting join, convert a communication link of the one or more communication links between the at least one build operator and the at least one probe operator into a synchronous link.
14. The system of claim 13 , wherein the synchronous link denotes a local one-to-one link between the at least one build operator and the at least one probe operator, and wherein the at least one probe operator performs the broadcasting join.
15. The system of claim 14 , wherein the processing device is to convert the communication link of the one or more communication links between the at least one build operator and the at least one probe operator from an initially inactive and adaptive state to the synchronous link.
16. The system of claim 12 , wherein to manage the state of the one or more communication links, the processing device is further to, in response to determining to join the plurality of relations via the re-partitioning join, convert a communication link of the one or more communication links between a build operator and a probe operator into a partition link.
17. The system of claim 12 , wherein the first relation and the second relation are partitioned among a plurality of processors of a distributed system.
18. The system of claim 12 , wherein to perform the re-partition join, the processing device is to:
send a partition of the second relation to a local probe operator instance as a single pointer to a block of memory or a file on a disk.
19. The system of claim 12 , wherein to perform a broadcasting join, the processing device is to:
forward a local partition of the first relation to a local instance of a probe operator for performing the relational join query.
20. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processing device, cause the processing device to:
receive a relational join query for a join operation comprising a predicate and a plurality of relations, wherein the plurality of relations comprises a first relation and a second relation;
determine, by the processing device, a type of join to be performed on the plurality of relations based on an actual size of the second relation and an estimated size of the first relation, wherein the type of join is a broadcasting join or a re-partitioning join; and
manage, by the processing device, a state of one or more communication links between at least one build operator and at least one probe operator based on the type of join to be performed to perform the join operation.
21. The non-transitory computer readable medium of claim 20 , wherein the relations are distributed over a plurality of processing nodes.
22. The non-transitory computer readable medium of claim 20 , wherein to manage the state of the one or more communication links, the processing device is to:
in response to determining to join the plurality of relations via the broadcasting join, convert a communication link of the one or more communication links between the at least one build operator and at least one probe operator into a synchronous link.
23. The non-transitory computer readable medium of claim 22 , wherein the synchronous link denotes a local one-to-one link between the at least one build operator and the at least one probe operator, and wherein the at least one probe operator performs the broadcasting join.
24. The non-transitory computer readable medium of claim 20 , wherein to manage the state of the one or more communication links, the processing device is to:
in response to determining to join the plurality of relations via the re-partitioning join, convert a communication link of the one or more communication links between the at least one build operator and the at least one probe operator into a partition link.
25. The non-transitory computer readable medium of claim 20 , wherein during a broadcasting join, a local partition of the first relation is forwarded to a local instance of the at least one probe operator for performing the relational join query.