Weighted auto-sharing
Methods, systems, and apparatus for automatic sharding and load balancing in a distributed data processing system. In one aspect, a method includes determining workload distribution for an application across worker computers and in response to determining a load balancing operation is required: selecting a first worker computer having a highest load measure relative to respective load measure of the other work computers; determining one or more move operations for a partition of data assigned to the first worker computer and a weight for each move operation; and selecting the move operation with a highest weight the selected move operation.
1. A computer-implemented method executed by data processing hardware that causes the data processing hardware to perform operations comprising:
partitioning a data set for an application job into a plurality of partitions based on a key;
assigning, to each worker computer in a set of worker computers, one or more partitions of the plurality of partitions;
receiving, from each respective worker computer in the set of worker computers, a respective load measure indicating a computational load of the respective worker computer;
determining, using the respective load measure of each respective worker computer, a weight of a move operation for moving at least one partition from a first worker computer of the set of worker computers to a second worker computer of the set of worker computers;
determining that the weight of the move operation satisfies a threshold; and
in response to determining that the weight of the move operation satisfies the threshold, moving the at least one partition from the first worker computer of the set of worker computers to the second worker computer of the set of worker computers.
2. The method of claim 1 , wherein the key comprises an atomic unit of work placement.
3. The method of claim 1 , wherein each worker computer in the set of worker computers receives a different one or more partitions of the plurality of partitions.
4. The method of claim 1 , wherein the operations further comprise determining an initial workload distribution for the application job to each worker computer in the set of worker computers.
5. The method of claim 1 , wherein the respective load measure of the first worker computer is higher than the respective load measure of each other worker computer in the set of worker computers.
6. The method of claim 1 , wherein the operations further comprise:
determining, for each partition, a constituent load measure for the partition;
determining pairs of adjacent partitions, each pair comprising two partitions that collectively have a contiguous range of key values; and
for each pair of adjacent partitions for which a sum of the constituent load measures of the partition does not meet a load measure merger threshold, merging the adjacent partitions into a single partition.
7. The method of claim 6 , wherein merging the adjacent partitions into a single partition occurs prior to receiving, from each respective worker computer in the set of worker computers, the respective load measure.
8. The method of claim 1 , wherein the respective load measure of the second worker computer is lower than the respective load measure of each other worker computer in the set of worker computers.
9. The method of claim 1 , wherein the first worker computer of the set of worker computers is assigned a different number of partitions than the second worker computer of the set of worker computers.
10. The method of claim 1 , wherein determining the weight of the move operation comprises determining a ratio of a benefit of the move operation to a cost of the move operation.
11. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
partitioning a data set for an application job into a plurality of partitions based on a key;
assigning, to each worker computer in a set of worker computers, one or more partitions of the plurality of partitions;
receiving, from each respective worker computer in the set of worker computers, a respective load measure indicating a computational load of the respective worker computer;
determining, using the respective load measure of each respective worker computer, a weight of a move operation for moving at least one partition from a first worker computer of the set of worker computers to a second worker computer of the set of worker computers;
determining that the weight of the move operation satisfies a threshold; and
in response to determining that the weight of the move operation satisfies the threshold, moving the at least one partition from the first worker computer of the set of worker computers to the second worker computer of the set of worker computers.
12. The system of claim 11 , wherein the key comprises an atomic unit of work placement.
13. The system of claim 11 , wherein each worker computer in the set of worker computers receives a different one or more partitions of the plurality of partitions.
14. The system of claim 11 , wherein the operations further comprise determining an initial workload distribution for the application job to each worker computer in the set of worker computers.
15. The system of claim 11 , wherein the respective load measure of the first worker computer is higher than the respective load measure of each other worker computer in the set of worker computers.
16. The system of claim 11 , wherein the operations further comprise:
determining, for each partition, a constituent load measure for the partition;
determining pairs of adjacent partitions, each pair comprising two partitions that collectively have a contiguous range of key values; and
for each pair of adjacent partitions for which a sum of the constituent load measures of the partition does not meet a load measure merger threshold, merging the adjacent partitions into a single partition.
17. The system of claim 16 , wherein merging the adjacent partitions into a single partition occurs prior to receiving, from each respective worker computer in the set of worker computers, the respective load measure.
18. The system of claim 11 , wherein the respective load measure of the second worker computer is lower than the respective load measure of each other worker computer in the set of worker computers.
19. The system of claim 11 , wherein the first worker computer of the set of worker computers is assigned a different number of partitions than the second worker computer of the set of worker computers.
20. The system of claim 11 , wherein determining the weight of the move operation comprises determining a ratio of a benefit of the move operation to a cost of the move operation.