Throughput-optimized, quality-of-service aware power capping system
This disclosure describes a method to minimize disruption for throughput oriented jobs in power oversubscription services with a dynamic control. The mechanism controls power in a hardware-agnostic way, and the policy employs a multi-threshold approach that balances power safety with workload impact. Moreover, an alternative control mechanism ensures proper system operation while power measurements are unavailable.
1. A method, comprising:
receiving, by one or more processors, power measurements for a plurality of machines in a datacenter, the plurality of machines executing one or more tasks;
identifying power limitations for the plurality of machines, including identifying a first threshold and identifying a second threshold higher than the first threshold;
determining whether to shed power consumed by the plurality of machines, comprising determining whether the received power measurements meet or exceed the second threshold;
commanding, by the one or more processors when the received power measurements meet or exceed the second threshold, the plurality of machines to operate in accordance with one or more restrictions to reduce power consumption below the first threshold;
determining, by the one or more processors, whether the received power measurements exceed the first threshold during a predetermined period of time after reducing power consumption below the first threshold; and
commanding the plurality of machines to reduce power below the first threshold when the first threshold is met or exceeded during the predetermined period of time.
2. The method of claim 1 ,
wherein commanding the plurality of machines to operate in accordance with one or more restrictions to reduce power consumption below the first threshold comprises sending a command to reduce power consumption by a first predetermined percentage.
3. The method of claim 2 , wherein when the first threshold is met or exceeded during the predetermined period of time, the method further comprises sending a second command to the plurality of machines to reduce power consumption by a second predetermined percentage that is lower than the first predetermined percentage.
4. The method of claim 1 , wherein the measurements are received from one or more power meters coupled to the plurality of machines.
5. The method of claim 1 , wherein commanding the plurality of machines to operate in accordance with the one or more restrictions comprises restricting processing time of a task inside a machine level scheduler for all the machines in a power domain.
6. The method of claim 5 , wherein restricting the processing time comprises applying a first multiplier for low priority tasks, wherein the first multiplier is selected to prevent the low priority tasks from running and consuming power.
7. The method of claim 6 , further comprising applying a second multiplier for the low priority tasks.
8. The method of claim 6 , further comprising exempting one or more high priority tasks from the limitations.
9. The method of claim 1 , further comprising limiting a number of available schedulable processor entities to control power when the power measurements become unavailable.
10. The method of claim 1 , wherein commanding the plurality of machines to operate in accordance with one or more restrictions comprises making a portion of processing components of each of the plurality of machines unavailable to tasks.
11. A system, comprising:
one or more processors in communication with a plurality of machines executing one or more tasks, the one or more processors configured to:
receive power measurements for the plurality of machines;
identify power limitations for the plurality of machines, including identifying a first threshold and identifying a second threshold higher than the first threshold;
determine whether to shed power consumed by the plurality of machines, comprising determining whether the received power measurements meet or exceed the second threshold;
command, by the one or more processors when the received power measurements meet or exceed the second threshold, the plurality of machines to operate in accordance with one or more restrictions to reduce power consumption below the first threshold;
determine whether the received power measurements exceed the first threshold during a predetermined period of time after reducing power consumption below the first threshold; and
command the plurality of machines to reduce power below the first threshold when the first threshold is met or exceeded during the predetermined period of time.
12. The system of claim 11 ,
wherein commanding the plurality of machines to operate in accordance with one or more restrictions to reduce power consumption below the first threshold comprises sending a command to reduce power consumption by a first predetermined percentage.
13. The system of claim 12 , wherein when the first threshold is met or exceeded during the predetermined period of time, the one or more processors are further configured to send a second command to the plurality of machines to reduce power consumption by a second predetermined percentage that is lower than the first predetermined percentage.
14. The system of claim 11 , wherein the measurements are received from one or more power meters coupled to the plurality of machines.
15. The system of claim 11 , wherein commanding the plurality of machines to operate in accordance with the one or more restrictions comprises restricting processing time of a task inside a machine level scheduler for all the machines in a power domain.
16. The system of claim 15 , wherein in restricting the processing time the one or more processors are configured to apply a first multiplier for low priority tasks, wherein the first multiplier is selected to prevent the low priority tasks from running and consuming power.
17. The system of claim 16 , wherein in restricting the processing time the one or more processors are configured to exempt one or more high priority tasks from the limitations.
18. The system of claim 11 , wherein commanding the plurality of machines to operate in accordance with one or more restrictions comprises making a portion of processing components of each of the plurality of machines unavailable to tasks.
19. A non-transitory computer-readable medium storing instructions executable by one or more processors for performing a method, comprising:
receiving power measurements for a plurality of machines in a datacenter, the plurality of machines executing one or more tasks;
identifying power limitations for the plurality of machines, including identifying a first threshold and identifying a second threshold higher than the first threshold;
determining whether to shed power consumed by the plurality of machines, comprising determining whether the received power measurements meet or exceed the second threshold;
commanding, when the received power measurements meet or exceed the second threshold, the plurality of machines to operate in accordance with one or more restrictions to reduce power consumption below the first threshold;
determine whether the received power measurements exceed the first threshold during a predetermined period of time after reducing power consumption below the first threshold; and
command the plurality of machines to reduce power below the first threshold when the first threshold is met or exceeded during the predetermined period of time.