Balancing computation and communication power in power constrained clusters
Systems, apparatuses, and methods for balancing computation and communication power in power constrained environments. A data processing cluster with a plurality of compute nodes may perform parallel processing of a workload in a power constrained environment. Nodes that finish tasks early may be power-gated based on one or more conditions. In some scenarios, a node may predict a wait duration and go into a reduced power consumption state if the wait duration is predicted to be greater than a threshold. The power saved by power-gating one or more nodes may be reassigned for use by other nodes. A cluster agent may be configured to reassign the unused power to the active nodes to expedite workload processing.
1. A system comprising: a plurality of nodes, each of the nodes comprising circuitry configured to: execute computing processes; calculate power consumption of the node; and store power consumption data of the node in a storage device; a cluster agent configured to: monitor a power usage of each of the plurality of nodes based at least in part on the power consumption data; track historical computation durations; assign a power budget to each node of the plurality of nodes; predict, based at least in part on the historical computation durations, how much performance would decrease on a first node due to an increased latency from having to wake from a power-gated state; and increase on a second node, if power were to be reassigned from the first node to the second node; if said increase in performance is determined to be greater than said decrease in performance: cause the first node to enter the power-gated state; and reassign power saved by power-gating the first node to the second node.
2. The system as recited in claim 1 , wherein predicting said decrease in performance on the first node comprises calculating a latency caused by having to wake the first node from the power-gated state, and wherein predicting said increase in performance on the second node comprises predicting a reduced amount of computation time if power is reassigned from the first node to the second node.
3. The system as recited in claim 1 , wherein each node of the plurality of nodes comprises a node agent configured to power-gate a corresponding node responsive to detecting one or more conditions.
4. The system as recited in claim 3 , wherein the one or more conditions comprise one or more of reaching a communication wait phase, predicting that an expected wait time is greater than a threshold, and determining the one or more other nodes are not within a threshold of their thermal design power.
5. The system as recited in claim 3 , wherein each node agent is further configured to predict an expected wait time responsive to a corresponding node finishing an assigned task.
6. The system as recited in claim 3 , wherein each node agent is further configured to poll a network for a message for a first period of time responsive to reaching a communication wait phase, and wherein the one or more conditions comprise the first period of time elapsing after reaching the communication wait phase.
7. A method comprising: assigning, by a cluster agent, a power budget to each node of a plurality of nodes of a computing system; calculating power consumption of each of the plurality of nodes; and storing power consumption data of each of the plurality of nodes in a storage device; monitoring a power usage of each of the plurality of nodes based at least in part on the power consumption data; tracking historical computation durations; predicting, based at least in part on the historical computation durations, how much performance would decrease on a first node due to an increased latency from having to wake from a power-gated state; and increase on a second node, if power were to be reassigned from the first node to the second node; if said increase in performance is determined to be greater than said decrease in performance: causing the first node to enter the power-gated state; and reassigning, by the cluster agent, power saved by power-gating the first node to the second node.
8. The method as recited in claim 7 , wherein predicting said decrease in performance on the first node comprises calculating a latency caused by having to wake the first node from the power-gated state, and wherein predicting said increase in performance on the second node comprises predicting a reduced amount of computation time if power is reassigned from the first node to the second node.
9. The method as recited in claim 7 , wherein each node of the plurality of nodes comprises a node agent, the method further comprising:
the node agent power-gating a corresponding node responsive to detecting one or more conditions.
10. The method as recited in claim 9 , wherein the one or more conditions comprise one or more of reaching a communication wait phase, predicting that an expected wait time is greater than a threshold, and determining the one or more other nodes are not within a threshold of their thermal design power.
11. The method as recited in claim 9 , further comprising:
predicting an expected wait time responsive to a corresponding node finishing an assigned task.
12. The method as recited in claim 9 , further comprising:
polling a network for a message for a first period of time responsive to reaching a communication wait phase, and wherein the one or more conditions comprise the first period of time elapsing after reaching the communication wait phase.
13. A non-transitory computer readable storage medium storing program instructions, wherein the program instructions are executable by a processor to: assign a power budget to each node of a plurality of nodes of a computing system; calculate power consumption of each of the plurality of nodes; store power consumption data of each of the plurality of nodes in a storage device; monitor a power usage of each of the plurality of nodes based at least in part on the power consumption data; track historical computation durations; predict, based at least in part on the historical computation durations, how much performance would decrease on a first node due to an increased latency from having to wake from a power-gated state; and increase on a second node, if power were to be reassigned from the first node to the second node; if said increase in performance is determined to be greater than said decrease in performance: cause the first node to enter the power-gated state; and reassign power saved by power-gating the first node to the second node.
14. The non-transitory computer readable storage medium as recited in claim 13 , wherein predicting said decrease in performance on the first node comprises calculating a latency caused by having to wake the first node from the power-gated state, and wherein predicting said increase in performance on the second node comprises predicting a reduced amount of computation time if power is reassigned from the first node to the second node.
15. The non-transitory computer readable storage medium as recited in claim 13 , wherein each node of the plurality of nodes comprises a node agent, and wherein the program instructions are further executable by a processor to power-gate a corresponding node responsive to detecting one or more conditions.
16. The non-transitory computer readable storage medium as recited in claim 15 , wherein the one or more conditions comprise one or more of reaching a communication wait phase, predicting that an expected wait time is greater than a threshold, and determining the one or more other nodes are not within a threshold of their thermal design power (TDP).
17. The non-transitory computer readable storage medium as recited in claim 15 , wherein the program instructions are further executable by a processor to predict an expected wait time responsive to a corresponding node finishing an assigned task.