Detecting data skew in a join operation
Systems, methods, and devices, for managing data skew during a join operation are disclosed. A method includes computing a hash value for a join operation and detecting data skew on a probe side of the join operation at a runtime of the join operation using a lightweight sketch data structure. The method includes identifying a frequent probe-side join key on the probe side of the join operation during a probe phase of the join operation. The method includes identifying a frequent build-side row having a build-side join key corresponding with the frequent probe-side join key. The method includes asynchronously distributing the frequent build-side row to one or more remote servers.
1. A method comprising:
probing, by a local server, a probe side table for data included in the probe side table that corresponds to any build keys included in a build side table, the probing performed as part of a join operation;
determining, by the local server, a frequency at which data is received from the probe side table during the probing;
in response to determining that the frequency meets or exceeds a threshold frequency, distributing, by the local server, data corresponding to a first build key to a remote server that transmitted the first build key more frequently than another remote server, the remote server processing the data corresponding to the first build key as part of the join operation; and
processing, by the local server, data corresponding to a second build key as part of the join operation.
2. The method of claim 1 , further comprising:
generating an output of the join operation based on the processing performed by the local server and the processing performed by the remote server.
3. The method of claim 2 , wherein the remote server is one of a plurality of remote servers to which data may be distributed for processing as part of the join operation.
4. The method of claim 3 , further comprising:
distributing, by the local server, data corresponding to a third build key to a second remote server from the plurality of remote servers, the second remote server processing the data corresponding to the third build key as part of the join operation, wherein the output of the join operation is further generated based on the processing performed by the second remote server.
5. The method of claim 4 , wherein distributing the data corresponding to the third build key to the second remote server is based on a determination that a frequency at which the third build key is probed meets or exceeds a predetermined threshold.
6. The method of claim 1 , wherein determining the frequency at which data is received from the probe side table during the probing comprises:
determining a frequency at which the first build key is probed during the probing.
7. The method of claim 6 , wherein determining the frequency at which data is received from the probe side table during the probing further comprises:
determination that the frequency at which the first build key is probed meets or exceeds a predetermined threshold, wherein distributing the data corresponding to the first build key to the remote server is based on the determination that the frequency at which the first build key is probed meets or exceeds a predetermined threshold.
8. The method of claim 1 , wherein probing the probe side table comprises:
computing, by the local server, a hash value for the join operation.
9. The method of claim 8 , wherein probing the probe side table further comprises:
selecting a rowset comprising a plurality of rows of the join operation; and
probing each of the plurality of rows of the rowset into a space saving algorithm using the hash value for the join operation.
10. The method of claim 9 , further comprising:
calculating a total number of rows of the join operation that have been probed into the space saving algorithm.
11. The method of claim 10 , further comprising:
calculating the threshold frequency based on the total number of rows of the join operation that have been probed into the space saving algorithm.
12. A local computing system comprising:
one or more computer processors; and
one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the local computing system to perform operations comprising:
probing a probe side table for data included in the probe side table that corresponds to any build keys included in a build side table, the probing performed as part of a join operation;
determining a frequency at which data is received from the probe side table during the probing;
in response to determining that the frequency meets or exceeds a threshold frequency, distributing data corresponding to a first build key to a remote server that transmitted the first build key more frequently than another remote server, the remote server processing the data corresponding to the first build key as part of the join operation; and
processing data corresponding to a second build key as part of the join operation.
13. The local computing system of claim 12 , the operations further comprising:
generating an output of the join operation based on the processing performed by the local computing system and the processing performed by the remote server.
14. The local computing system of claim 13 , wherein the remote server is one of a plurality of remote servers to which data may be distributed for processing as part of the join operation.
15. The local computing system of claim 14 , the operations further comprising:
distributing data corresponding to a third build key to a second remote server from the plurality of remote servers, the second remote server processing the data corresponding to the third build key as part of the join operation, wherein the output of the join operation is further generated based on the processing performed by the second remote server.
16. The local computing system of claim 15 , wherein distributing the data corresponding to the third build key to the second remote server is based on a determination that a frequency at which the third build key is probed meets or exceeds a predetermined threshold.
17. The local computing system of claim 12 , wherein determining the frequency at which data is received from the probe side table during the probing comprises:
determining a frequency at which the first build key is probed during the probing.
18. The local computing system of claim 17 , wherein determining the frequency at which data is received from the probe side table during the probing further comprises:
determination that the frequency at which the first build key is probed meets or exceeds a predetermined threshold, wherein distributing the data corresponding to the first build key to the remote server is based on the determination that the frequency at which the first build key is probed meets or exceeds a predetermined threshold.
19. The local computing system of claim 12 , wherein probing the probe side table comprises:
computing a hash value for the join operation.
20. The local computing system of claim 19 , wherein probing the probe side table further comprises:
selecting a rowset comprising a plurality of rows of the join operation; and
probing each of the plurality of rows of the rowset into a space saving algorithm using the hash value for the join operation.
21. The local computing system of claim 20 , the operations further comprising:
calculating a total number of rows of the join operation that have been probed into the space saving algorithm.
22. The local computing system of claim 21 , the operations further comprising:
calculating the threshold frequency based on the total number of rows of the join operation that have been probed into the space saving algorithm.
23. A non-transitory computer-readable medium storing instructions that, when executed by one or more local servers, cause the one or more local servers to perform operations comprising:
probing a probe side table for data included in the probe side table that corresponds to any build keys included in a build side table, the probing performed as part of a join operation;
determining a frequency at which data is received from the probe side table during the probing;
in response to determining that the frequency meets or exceeds a threshold frequency, distributing, by the local server, data corresponding to a first build key to a remote server that transmitted the first build key more frequently than another remote server, the remote server processing the data corresponding to the first build key as part of the join operation; and
processing data corresponding to a second build key as part of the join operation.
24. The non-transitory computer-readable medium of claim 23 , the operations further comprising:
generating an output of the join operation based on the processing performed by the one or more local servers and the processing performed by the remote server.
25. The non-transitory computer-readable medium of claim 24 , wherein the remote server is one of a plurality of remote servers to which data may be distributed for processing as part of the join operation.
26. The non-transitory computer-readable medium of claim 25 , the operations further comprising:
distributing data corresponding to a third build key to a second remote server from the plurality of remote servers, the second remote server processing the data corresponding to the third build key as part of the join operation, wherein the output of the join operation is further generated based on the processing performed by the second remote server.
27. The non-transitory computer-readable medium of claim 26 , wherein distributing the data corresponding to the third build key to the second remote server is based on a determination that a frequency at which the third build key is probed meets or exceeds a predetermined threshold.
28. The non-transitory computer-readable medium of claim 23 , wherein determining the frequency at which data is received from the probe side table during the probing comprises:
determining a frequency at which the first build key is probed during the probing.
29. The non-transitory computer-readable medium of claim 28 , wherein determining the frequency at which data is received from the probe side table during the probing further comprises:
determination that the frequency at which the first build key is probed meets or exceeds a predetermined threshold, wherein distributing the data corresponding to the first build key to the remote server is based on the determination that the frequency at which the first build key is probed meets or exceeds a predetermined threshold.
30. The non-transitory computer-readable medium of claim 23 , wherein probing the probe side table comprises:
computing, by the local server, a hash value for the join operation.