Systems and methods for dynamic partitioning in distributed environments

1. A computer-implemented method for dynamic partitioning in distributed computing environments, the method comprising:

receiving, at a processor, a first data set and a second data set;

mapping, by the processor, the first data set into a first set of key-value pairs;

mapping, by the processor, the second data set into a second set of key-value pairs;

estimating, by the processor using a sketch, a frequency count for each key based on the first set of key-value pairs and the second set of key-value pairs;

determining, by the processor, whether the estimated frequency count for each key is greater than or equal to a predetermined threshold;

partitioning, by the processor, the key when the estimated frequency count for the key is greater than or equal to the predetermined threshold;

retrieving, by the processor from a master node, a global frequency count for each key of the key-value pairs; and

determining, by the processor, an updated frequency count for each key based on the global frequency count for each key and the estimated frequency count for each key.

2. The method of claim 1 , further comprising:

transmitting, by the processor to the master node, the updated frequency count for each key.

3. The method of claim 1 , wherein determining whether the frequency count for each key is greater than or equal to the predetermined threshold is based on the updated frequency count for each key.

4. The method of claim 1 , wherein the sketch is one of a lossy algorithm and a count-min sketch.

5. The method of claim 1 , further comprising:

grouping, by the processor, the values of the key-value pairs by the key to form a set of values grouped by each key.

6. The method of claim 4 , further comprising:

reducing, by the processor, the values of each set of values grouped by each key into a set of pairs.

7. A system for dynamic partitioning in distributed computing environments, the system including:

a data storage device that stores instructions for dynamic partitioning in distributed computing environments; and

a processor configured to execute the instructions to perform a method including:

receiving a first data set and a second data set;

mapping the first data set into a first set of key-value pairs;

mapping the second data set into a second set of key-value pairs;

estimating, using a sketch, a frequency count for each key based on the first set of key-value pairs and the second set of key-value pairs;

determining whether the estimated frequency count for each key is greater than or equal to a predetermined threshold;

partitioning the key when the estimated frequency count for the key is greater than or equal to the predetermined threshold;

retrieving, from a master node, a global frequency count for each key of the key-value pairs; and

determining an updated frequency count for each key based on the global frequency count for each key and the estimated frequency count for each key.

8. The system of claim 7 , wherein the processor is further configured to execute the instructions to perform the method including:

transmitting, to the master node, the updated frequency count for each key.

9. The system of claim 7 , wherein determining whether the frequency count for each key is greater than or equal to the predetermined threshold is based on the updated frequency count for each key.

10. The system of claim 7 , wherein the sketch is one of a lossy algorithm and a count-min sketch.

11. The system of claim 7 , wherein the processor is further configured to execute the instructions to perform the method including:

grouping the values of the key-value pairs by the key to form a set of values grouped by each key.

12. The system of claim 11 , wherein the processor is further configured to execute the instructions to perform the method including:

reducing the values of each set of values grouped by each key into a set of pairs.

13. A non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for dynamic partitioning in distributed computing environments, the method including:

receiving a first data set and a second data set;

mapping the first data set into a first set of key-value pairs;

mapping the second data set into a second set of key-value pairs;

estimating, using a sketch, a frequency count for each key based on the first set of key-value pairs and the second set of key-value pairs;

determining whether the estimated frequency count for each key is greater than or equal to a predetermined threshold;

partitioning the key when the estimated frequency count for the key is greater than or equal to the predetermined threshold;

retrieving, from a master node, a global frequency count for each key of the key-value pairs; and

determining an updated frequency count for each key based on the global frequency count for each key and the estimated frequency count for each key.

14. The computer-readable medium of claim 13 , further comprising:

transmitting, to the master node, the updated frequency count for each key.

15. The computer-readable medium of claim 13 , wherein determining whether the frequency count for each key is greater than or equal to the predetermined threshold is based on the updated frequency count for each key.

16. The computer-readable medium of claim 13 , wherein the sketch is one of a lossy algorithm and a count-min sketch.

17. The computer-readable medium of claim 13 , further comprising:

grouping the values of the key-value pairs by the key to form a set of values grouped by each key.