Adaptive distribution for hash operations
A method, apparatus, and system for join operations of a plurality of relations that are distributed over a plurality of storage locations over a network of computing components. An example method includes: receiving a relational join query for a join operation associated with a first relation and a second relation; generating at least one build operator and at least one probe operator to perform build operations and probe operations, respectively, of the join operation; and managing a state of one or more communication links between the at least one build operator and the at least one probe operator based on a size of the second relation as determined by the at least one build operator and an estimated size of the first relation.
1. A method of joining relations, the method comprising:
receiving a relational join query for a join operation associated with a first relation and a second relation;
generating, by a processing device, at least one build operator and at least one probe operator to perform build operations and probe operations, respectively, of the join operation;
converting, by the processing device, a state of one or more communication links between the at least one build operator and the at least one probe operator to one of a plurality of states, wherein the converting is based on a size of the second relation as determined by the at least one build operator and an estimated size of the first relation, and wherein the plurality of states comprise a broadcast link state and a synchronous link state; and
performing the join operation associated with the first relation and the second relation utilizing the one or more communication links between the at least one build operator and the at least one probe operator.
2. The method of claim 1 , further comprising:
determining, by the processing device, a type of the join operation associated with the first relation and the second relation based on the size of the second relation as determined by the at least one build operator and the estimated size of the first relation.
3. The method of claim 2 , wherein the type of the join operation associated with the first relation and the second relation comprises a broadcasting join or a re-partitioning join.
4. The method of claim 2 , wherein determining the type of the join operation associated with the first relation and the second relation comprises:
determining the type of the join operation associated with the first relation and the second relation as a re-partitioning join; and
converting the state of the one or more communication links between the at least one build operator and the at least one probe operator into the synchronous link state.
5. The method of claim 4 , wherein the synchronous link state denotes a local one-to-one link between the at least one build operator and the at least one probe operator, and
wherein the at least one probe operator performs the re-partitioning join.
6. The method of claim 2 , wherein determining the type of the join operation associated with the first relation and the second relation comprises:
determining the type of the join operation associated with the first relation and the second relation as a broadcasting join; and
converting the state of the one or more communication links between the at least one build operator and the at least one probe operator into a partition the broadcast link state.
7. The method of claim 4 , further comprising, during the re-partitioning join, sending a partition of the second relation to a local probe operator instance as a single pointer to a block of memory or a file on a disk.
8. The method of claim 1 , wherein converting the state of the one or more communication links comprises converting the state of the one or more communication links between the at least one build operator and the at least one probe operator from an initially inactive and adaptive state to the broadcast link state.
9. The method of claim 1 , wherein converting the state of the one or more communication links comprises converting the state of the one or more communication links between the at least one build operator and the at least one probe operator from an initially inactive and adaptive state to the synchronous link state.
10. The method of claim 1 , further comprising:
generating a hash table of the first relation; and
partitioning the first relation to the at least one build operator based on the hash table of the first relation.
11. A system comprising:
a memory; and
a processing device, operatively coupled to the memory, the processing device to:
receive a relational join query for a join operation associated with a first relation and a second relation;
generate at least one build operator and at least one probe operator to perform build operations and probe operations, respectively, of the join operation;
convert a state of one or more communication links between the at least one build operator and the at least one probe operator to one of a plurality of states, wherein the converting is based on a size of the second relation as determined by the at least one build operator and an estimated size of the first relation, and wherein the plurality of states comprise a broadcast link state and a synchronous link state; and
perform the join operation associated with the first relation and the second relation utilizing the one or more communication links between the at least one build operator and the at least one probe operator.
12. The system of claim 11 , wherein the processing device is further to:
determine a type of the join operation associated with the first relation and the second relation based on the size of the second relation as determined by the at least one build operator and the estimated size of the first relation.
13. The system of claim 12 , wherein the type of the join operation associated with the first relation and the second relation comprises a broadcasting join or a re-partitioning join.
14. The system of claim 12 , wherein to determine the type of the join operation associated with the first relation and the second relation the processing device is to:
determine the type of the join operation associated with the first relation and the second relation as a re-partitioning join; and
convert the state of the one or more communication links between the at least one build operator and the at least one probe operator into the synchronous link state.
15. The system of claim 12 , wherein to determine the type of the join operation associated with the first relation and the second relation the processing device is to:
determine the type of the join operation associated with the first relation and the second relation as a broadcasting join; and
convert the state of the one or more communication links between the at least one build operator and the at least one probe operator into the broadcast link state.
16. The system of claim 11 , wherein to convert the state of the one or more communication links the processing device is to convert the state of the one or more communication links between the at least one build operator and the at least one probe operator from an initially inactive and adaptive state to the broadcast link state or to the synchronous link state.
17. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processing device, cause the processing device to:
receive a relational join query for a join operation associated with a first relation and a second relation;
generate at least one build operator and at least one probe operator to perform build operations and probe operations, respectively, of the join operation;
convert a state of one or more communication links between the at least one build operator and the at least one probe operator to one of a plurality of states, wherein the converting is based on a size of the second relation as determined by the at least one build operator and an estimated size of the first relation, and wherein the plurality of states comprise a broadcast link state and a synchronous link state; and
perform the join operation associated with the first relation and the second relation utilizing the one or more communication links between the at least one build operator and the at least one probe operator.
18. The non-transitory computer readable medium of claim 17 , wherein the processing device is further to:
determine a type of the join operation associated with the first relation and the second relation based on the size of the second relation as determined by the at least one build operator and the estimated size of the first relation.
19. The non-transitory computer readable medium of claim 18 , wherein to determine the type of the join operation associated with the first relation and the second relation the processing device is to:
determine the type of the join operation associated with the first relation and the second relation as a re-partitioning join; and
convert the state of the one or more communication links between the at least one build operator and the at least one probe operator into the synchronous link.
20. The non-transitory computer readable medium of claim 18 , wherein to determine the type of the join operation associated with the first relation and the second relation the processing device is to:
determine the type of the join operation associated with the first relation and the second relation as a broadcasting join; and
convert the state of the one or more communication links between the at least one build operator and the at least one probe operator into the broadcast link state.