Managing cluster-level performance variability without a centralized controller
Systems, apparatuses, and methods for managing cluster-level performance variability without a centralized controller are described. Each node of a multi-node cluster tracks a maximum and minimum progress across the plurality of nodes for a workload executed by the cluster. Each node also tracks its local progress on its current task. Each node also utilizes a comparison of the local progress to reported maximum and minimum progress across the cluster to identify a critical, or slow, node and whether to increase or reduce an amount of power allocated to the node. The nodes append information about the maximum and minimum progress to messages sent to other nodes to report their knowledge of maximum and minimum progress with other nodes. A node updates its local information if the node receives a message from another node with more up-to-date information about the state of progress across the cluster.
1. A system comprising:
a computing network;
a plurality of nodes interconnected via the computing network; and
a cluster agent configured to map tasks of a workload to the plurality of nodes;
wherein each node of the plurality of nodes is configured to:
receive maximum and minimum progress indications from one or more nodes of the plurality of nodes for a workload;
track a local progress of the node on a current task of the workload;
utilize a comparison of the local progress to the received maximum and minimum progress indications to determine whether to change performance parameters of the node; and
increase a performance of the node by changing said performance parameters to cause an increase in an amount of power allocated to the node, responsive to determining the local progress of the node is within a first threshold of the minimum progress.
2. The system as recited in claim 1 , wherein each node of the plurality of nodes is further configured to:
maintain a table of maximum and the minimum progress indications, wherein the table identifies a fastest node and a slowest node of the plurality of nodes for the workload; and
include information from the table in messages that are sent to one or more other nodes.
3. The system as recited in claim 2 , wherein each node of the plurality of nodes is further configured to send the table to either:
one or more randomly selected nodes of the plurality of nodes; or
one or more predetermined nodes of the plurality of nodes.
4. The system as recited in claim 1 , wherein responsive to determining a local progress of the node is within a second threshold of the maximum progress, each node of the plurality of nodes is configured to decrease an amount of power allocated to the node by changing said performance parameters.
5. The system as recited in claim 1 , wherein each node of the plurality of nodes is further configured to:
receive a first table from a first node;
compare the first table to a local table maintained by the node;
modify data in the local table to update a maximum progress indication in the local table responsive to determining a maximum progress indication in the first table is greater than the maximum progress indication in the local table; and
modify data in the local table to update a minimum progress indication in the local table responsive to determining:
a minimum progress indication in the first table is less than the minimum progress of the local table; and
an identity of a slowest node in the local table is different from an identity of a slowest node in the first table.
6. The system as recited in claim 5 , wherein each node of the plurality of nodes is further configured to:
copy an identity of a fastest node from the first table to the local table responsive to determining a maximum progress indication in the first table is greater than the maximum progress indication in the local table and responsive to determining an identity of a fastest node in the local table is different from an identity of the fastest node in the first table; and
copy an identity of a slowest node from the first table to the local table responsive to determining a minimum progress indication in the first table is less than the minimum progress indication in the local table and responsive to determining an identity of a slowest node in the local table is different from an identity of the slowest node in the first table.
7. A method comprising:
receiving, by a node in a computing system, maximum and minimum progress indications corresponding to a plurality of nodes in the computing system for a workload;
tracking, by the node, a local progress of the node on a current task of the workload;
utilizing a comparison of the local progress to the maximum and the minimum progress indications to determine whether to change performance parameters of the node; and
increasing a performance of the node by changing said performance parameters to cause an increase in an amount of power allocated to the node, responsive to determining the local progress of the node is within a first threshold of the minimum progress.
8. The method as recited in claim 7 , further comprising:
maintaining a table of the maximum and the minimum progress indications, wherein the table identifies a fastest node and a slowest node of the plurality of nodes for the workload; and
including information from the table in messages that are sent to one or more other nodes.
9. The method as recited in claim 8 , further comprising sending the table to either:
one or more randomly selected nodes of the plurality of nodes; or
one or more predetermined nodes of the plurality of nodes.
10. The method as recited in claim 7 , wherein changing said performance parameters comprises decreasing an amount of power allocated to the node responsive to determining a local progress of the node is within a second threshold of the maximum progress.
11. The method as recited in claim 7 , further comprising:
receiving a first table from a first node;
comparing the first table to a local table maintained by the node;
modifying data in the local table to update the maximum progress indication in the local table responsive to determining a maximum progress in the first table is greater than the maximum progress indication in the local table; and
modifying data in the local table to update the minimum progress indication in the local table responsive to determining a minimum progress indication in the first table is less than the minimum progress indication in the local table and responsive to determining an identity of a slowest node in the local table is different from an identity of a slowest node in the first table.
12. The method as recited in claim 11 , further comprising:
copying an identity of a fastest node from the first table to the local table responsive to determining a maximum progress indication in the first table is greater than the maximum progress indication in the local table and responsive to determining an identity of a fastest node in the local table is different from an identity of the fastest node in the first table; and
copying an identity of a slowest node from the first table to the local table responsive to determining a minimum progress indication in the first table is less than the minimum progress indication in the local table and responsive to determining an identity of a slowest node in the local table is different from an identity of the slowest node in the first table.
13. A non-transitory computer readable storage medium comprising program instructions, wherein the program instructions are executable to:
receive maximum and minimum progress indications from one or more nodes of the plurality of nodes for a workload;
track a local progress of the node on a current task of the workload;
utilize a comparison of the local progress to the received maximum and minimum progress indications to determine whether to change performance parameters of the node; and
increase a performance of the node by changing said performance parameters to cause an increase in an amount of power allocated to the node, responsive to determining the local progress of the node is within a first threshold of the minimum progress.
14. The non-transitory computer readable storage medium as recited in claim 13 , wherein the program instructions are further executable to:
maintain a table of the maximum and the minimum progress indications corresponding to the plurality of nodes, wherein the table identifies a fastest node and a slowest node of the plurality of nodes for the workload; and
include information from the table in messages that are sent to one or more other nodes.
15. The non-transitory computer readable storage medium as recited in claim 14 , wherein the program instructions are further executable to send the table to either:
one or more randomly selected nodes of the plurality of nodes; or
one or more predetermined nodes of the plurality of nodes.
16. The non-transitory computer readable storage medium as recited in claim 13 , wherein changing said performance parameters comprises decreasing an amount of power allocated to the node responsive to determining a local progress of the node is within a second threshold of the maximum progress.
17. The non-transitory computer readable storage medium as recited in claim 13 , wherein the program instructions are further executable to:
receive a first table from a first node;
compare the first table to a local table maintained by the node;
modify data in the local table to update the maximum progress indication in the local table responsive to determining a maximum progress indication in the first table is greater than the maximum progress indication in the local table; and
modify data in the local table to update the minimum progress of the local table responsive to determining a minimum progress indication in the first table is less than the minimum progress indication in the local table and responsive to determining an identity of a slowest node in the local table is different from an identity of a slowest node in the first table.