Datacenter level power management with reactive power capping
Disclosed techniques relate to managing power within a power distribution system. Power consumption corresponding to devices (e.g., servers) that receive power from an upstream device (e.g., a bus bar) may be monitored (e.g., by a service) to determine when power consumption corresponding to those devices has breached or will likely breach a budgeted threshold corresponding to an amount of power allocated to the upstream device. If the budgeted threshold is breached, or is likely to be breached, the service may initiate operations to distribute power caps for the devices and to initiate a timer. Although distributed, the power caps may be ignored by the devices until they are instructed to enforce the power caps (e.g., upon expiration of the timer). This allows the power consumption of the devices to exceed the budgeted power assigned to the upstream device at least until expiration of the timer while avoiding power outage events.
1. A method, comprising:
identifying, by a power management service, a plurality of components of a power system that are arranged according to a power distribution hierarchy, the power distribution hierarchy comprising a plurality of nodes organized according to respective levels of a plurality of levels;
monitoring, by the power management service, power consumption of a set of lower-level components, the set of lower-level components being represented by a subset of nodes of a first level of the power distribution hierarchy;
obtaining, by the power management service, a machine-learning model that has been previously trained using supervised learning and labeled training data examples to predict a likelihood that a collective power consumption of one or more lower-level components breaches a corresponding budget threshold of a corresponding higher-level component, a labeled training data example comprising 1) historical consumption data corresponding to a set of low-level components to which power has been passed through a respective higher-level component and 2) a label that indicates a corresponding likelihood that the collective power consumption of the set of lower-level components breaches a respective budget threshold of the respective higher-level component;
determining, by the power management service based at least in part on output obtained from the machine-learning model, that the power consumption of the set of lower-level components will likely breach a budget threshold associated with a higher-level component corresponding to a second level of the power distribution hierarchy, the second level being higher in the power distribution hierarchy than the first level; and
transmitting, by the power management service, a power cap value and a timing value with which a timer is initiated, a lower-level component of the set of lower-level components storing the power cap value in memory while allowing a respective power consumption of the lower-level component to exceed the power cap value, until the timer expires, indicating that a time period corresponding to the timing value has elapsed.
2. The method of claim 1 , wherein determining that the power consumption of the set of lower-level components will likely breach the budget threshold associated with the higher-level component corresponding to the second level of the power distribution hierarchy further comprises determining a predicted power consumption of the set of lower-level components that is likely to occur in a future time period.
3. The method of claim 1 , wherein the output received from the machine-learning model indicates an amount by which the power consumption corresponding to the set of lower-level components is expected to change.
4. The method of claim 1 , wherein storing the power cap value in the memory while allowing the respective power consumption of the lower-level component to exceed the power cap value causes the set of one or more lower-level components to breach the budget threshold associated with the higher-level component for at least a portion of the time period corresponding to the timing value.
5. A system, comprising:
one or more processors; and
memory configured to store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to:
identify a plurality of components that are arranged according to a power distribution hierarchy, the power distribution hierarchy comprising a plurality of nodes organized according to respective levels of a plurality of levels;
monitor power consumption of a set of lower-level components represented by a subset of nodes of a first level of the power distribution hierarchy, the set of lower-level components receiving power through a higher-level component corresponding to a second level of the power distribution hierarchy, the second level being higher in the power distribution hierarchy than the first level;
obtain a machine-learning model that has been previously trained using supervised learning and labeled training data examples to predict a likelihood that a collective power consumption of one or more lower-level components breaches a corresponding budget threshold of a corresponding higher-level component, a labeled training data example comprising 1) historical consumption data corresponding to a set of low-level components to which power has been passed through a respective higher-level component and 2) a label that indicates a corresponding likelihood that the collective power consumption of the set of lower-level components breaches a respective budget threshold of the respective higher-level component;
determine, based at least in part on output from the machine-learning model, that aggregate power consumption of the set of lower-level components will likely breach a budget threshold associated with the higher-level component; and
transmit a power cap value and a timing value with which a timer is initiated, wherein transmitting the power cap value and the timing value causes the set of lower-level components to breach the budget threshold until the timer expires indicating that a time period corresponding to the timing value has elapsed.
6. The system of claim 5 , wherein expiration of the timer triggers at least one lower-level component of the set of lower-level components to limit its respective power consumption based at least in part on the power cap value.
7. The system of claim 6 , wherein limiting the respective power consumption based at least in part on the power cap value is based at least in part on at least one of: dynamic frequency scaling or dynamic voltage scaling.
8. The system of claim 5 , wherein the higher-level component is a first higher-level component, wherein the second level of the power distribution hierarchy comprises a second higher-level component different from the first higher-level component, and wherein executing the instructions further causes the one or more processors to at least:
determine that there is an unutilized portion of power provisioned to the second higher-level component;
modify power consumption of a second lower-level component that is receiving power through the second higher-level component; and
transmit a cancellation signal that causes the timer to be canceled, wherein canceling the timer causes the lower-level component of the set of lower-level components to refrain from enforcing the power cap value.
9. The system of claim 5 , wherein the output further comprises a predicted aggregate power consumption for the set of lower-level components during a future time period, and wherein executing the instructions that determine that the aggregate power consumption of the set of lower-level components will likely breach the budget threshold associated with the higher-level component further causes the one or more processors to
compare the predicted aggregate power consumption for the set of lower-level components during the future time period to the budget threshold associated with the higher-level component.
10. The system of claim 5 , wherein the power cap value is determined based at least in part on determining an amount by which power consumption of the set of lower-level components will likely breach the budget threshold associated with the higher-level component.
11. A method, comprising:
monitoring, by a power management service, power consumption of a set of lower-level components of a power distribution system, the set of lower-level components receiving power through a higher-level component of the power distribution system;
determining, by the power management service and based at least in part on the monitoring, that an aggregate power consumption of the set of lower-level components will likely breach a budgeted power threshold associated with the higher-level component; and
responsive to determining that the aggregate power consumption of the set of lower-level components will likely breach the budgeted power threshold associated with the higher-level component, initiating a timer corresponding to a time period, wherein initiating the timer delays enforcement of a power cap of a lower-level component of the set of lower-level components for at least the time period corresponding to the timer, wherein the timer is initiated with a timing value that is determined based at least in part on at least one of: a rate of change of the aggregate power consumption of the set of lower-level components, a direction of change corresponding to the rate of change of the aggregate power consumption of the set of lower-level components, a tolerance to a power circuit associated with the higher-level component, or an expected time needed to initiate the power capping at each component in the set of lower-level components.
12. The method of claim 11 , wherein initiating the timer corresponding to the timing period further comprises transmitting, by the power management service, the timing value to an intermediate component from which the lower-level component receives power, the intermediate component receiving power from the higher-level component, wherein transmitting the timing value causes the intermediate component to generate the timer corresponding to the timing value, the timer being configured to expire based on expiration of the time period corresponding to the timing value.
13. The method of claim 12 , wherein the intermediate component triggers enforcement of the power cap on the lower-level component upon expiration of the timer.
14. A system, comprising:
one or more processors; and
memory comprising computer-executable instructions that, when executed by the one or more processors, causes the one or more processors to:
monitor power consumption of a set of downstream components of a power distribution system, the set of downstream components receiving power through an upstream component of the power distribution system;
determine, based at least in part on the power consumption, that an aggregate power consumption of the set of downstream components will breach a budgeted power threshold at a future time period, the budgeted power threshold being associated with the upstream component; and
responsive to determining that the aggregate power consumption of the set of downstream components will likely breach the budgeted power threshold associated with the upstream component, initiate a timer corresponding to a time period, wherein initiating the timer delays enforcement of a power cap of a downstream component of the set of downstream components for at least the time period corresponding to the timer, wherein the timer is initiated with a timing value that is determined based at least in part on at least one of: a rate of change of the aggregate power consumption of the set of downstream components, a direction of change corresponding to the rate of change of the aggregate power consumption of the set of downstream components, a tolerance to a power circuit associated with the upstream component, or an expected time needed to initiate power capping at each component in the set of downstream components.
15. The system of claim 14 , wherein executing the computer-executable instructions further causes the one or more processors to:
train a machine-learning model to identify a likelihood value that indicates a likelihood that corresponding aggregated power consumption of one or more downstream components will breach a power threshold of a corresponding upstream component;
provide historical power consumption data of the set of downstream components to the machine-learning model as input; and
receive output from the machine-learning model, the output indicating a corresponding likelihood that the corresponding aggregate power consumption of the set of downstream components will breach the budgeted power threshold associated with the upstream component.
16. The system of claim 14 , wherein executing the computer-executable instructions that initiate the timer further causes the one or more processors to transmit the timing value corresponding to the time period and a power cap value corresponding to the power cap, the timing value being used by an intermediate component to generate the timer, the power cap value being stored in memory of the downstream component of the set of downstream components while the enforcement of the power cap is delayed.