Systems and methods for dynamic partitioning in distributed environments

1. A computer-implemented method for dynamic partitioning in distributed computing environments, the method comprising:

receiving, at a processor, a first data set and a second data set;

mapping, by the processor, the first data set into a first set of key-value pairs;

mapping, by the processor, the second data set into a second set of key-value pairs;

estimating, by the processor using a sketch, a frequency count for each key based on the first set of key-value pairs and the second set of key-value pairs;

retrieving, by the processor from a master node, a global frequency count for each key of the key-value pairs;

determining, by the processor, an updated frequency count for each key based on the global frequency count for each key and the estimated frequency count for each key;

determining, by the processor, whether the estimated frequency count for each key is greater than or equal to a predetermined threshold based on the updated frequency count for each key; and

partitioning, by the processor, the key when the estimated frequency count for the key is greater than or equal to the predetermined threshold.

2. The method of claim 1 , further comprising:

transmitting, by the processor to the master node, the updated frequency count for each key.

3. The method of claim 1 , wherein the sketch is one of a lossy algorithm and a count-min sketch.

4. The method of claim 3 , further comprising:

reducing, by the processor, the values of each set of values grouped by each key into a set of pairs.

5. The method of claim 1 , further comprising:

grouping, by the processor, the values of the key-value pairs by the key to form a set of values grouped by each key.

6. A system for dynamic partitioning in distributed computing environments, the system including:

a data storage device that stores instructions for dynamic partitioning in distributed computing environments; and

a processor configured to execute the instructions to perform a method including:

receiving a first data set and a second data set;

mapping the first data set into a first set of key-value pairs;

mapping the second data set into a second set of key-value pairs;

estimating, using a sketch, a frequency count for each key based on the first set of key-value pairs and the second set of key-value pairs;

retrieving, by the processor from a master node, a global frequency count for each key of the key-value pairs;

determining, by the processor, an updated frequency count for each key based on the global frequency count for each key and the estimated frequency count for each key;

determining, by the processor, whether the estimated frequency count for each key is greater than or equal to a predetermined threshold based on the updated frequency count for each key; and

partitioning, by the processor, the key when the estimated frequency count for the key is greater than or equal to the predetermined threshold.

7. The system of claim 6 , wherein the processor is further configured to execute the instructions to perform the method including:

transmitting, to the master node, the updated frequency count for each key.

8. The system of claim 6 , wherein the sketch is one of a lossy algorithm and a count-min sketch.

9. The system of claim 6 , wherein the processor is further configured to execute the instructions to perform the method including:

grouping the values of the key-value pairs by the key to form a set of values grouped by each key.

10. The system of claim 9 , wherein the processor is further configured to execute the instructions to perform the method including:

reducing the values of each set of values grouped by each key into a set of pairs.

11. A non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for dynamic partitioning in distributed computing environments, the method including:

receiving a first data set and a second data set;

mapping the first data set into a first set of key-value pairs;

mapping the second data set into a second set of key-value pairs;

estimating, using a sketch, a frequency count for each key based on the first set of key-value pairs and the second set of key-value pairs;

retrieving, by the processor from a master node, a global frequency count for each key of the key-value pairs;

determining, by the processor, an updated frequency count for each key based on the global frequency count for each key and the estimated frequency count for each key;

determining, by the processor, whether the estimated frequency count for each key is greater than or equal to a predetermined threshold based on the updated frequency count for each key; and

partitioning, by the processor, the key when the estimated frequency count for the key is greater than or equal to the predetermined threshold.

12. The computer-readable medium of claim 11 , further comprising:

transmitting, to the master node, the updated frequency count for each key.

13. The computer-readable medium of claim 11 , wherein the sketch is one of a lossy algorithm and a count-min sketch.

14. The computer-readable medium of claim 11 , further comprising:

grouping the values of the key-value pairs by the key to form a set of values grouped by each key.