Microservice throttling based on learned demand predictions
A dynamically-managed microservices platform. The microservices platform is configured to selectively accept admission of microservices and to selectively throttle microservices based on a continually-trained machine learning model. The system is configured to implement multiple microservice lifecycle strategies, where individual ones of the multiple microservice lifecycle strategies use a prediction model having long-term and short term demand predictions. One or another of the multiple microservice lifecycle strategies are invoked at different times under different conditions. Under a first set of conditions, a prediction model is used to select a first strategy to admit or throttle a microservice based upon long-term or short-term demand predictions. Under a second set of conditions, the prediction model is used to select a different strategy to admit or throttle a microservice. The prediction model is continually trained such that new sets of observations and new sets of predicted outputs are codified into the machine learning model.
1. A non-transitory computer readable medium having stored thereon a sequence of instructions which, when stored in memory and executed by a processor cause acts comprising:
maintaining a microservices platform that responds to one or more throttling commands, wherein multiple throttling strategies are individually selectable by the microservices platform, individual ones of the multiple throttling strategies use at least one prediction model that outputs long-term and short-term demand predictions;
in response to receiving a first microservices throttling command at a first time under first conditions, using the at least one prediction model to select a first throttling strategy of the multiple throttling strategies to use to throttle a microservice, the first throttling strategy being selected based upon a first long-term or short-term demand prediction; and
in response to receiving a second microservices throttling command at a second time under second conditions, using the at least one prediction model to select a second throttling strategy of the multiple throttling strategies to use to throttle the microservice, the second throttling strategy being selected based upon a second long-term or short-term demand prediction,
wherein the first and second conditions are different, the first and second long-term or short-term demand predictions are different, and the first and second strategies are different.
2. The non-transitory computer readable medium of claim 1 , wherein respective long-term demand predictions correspond to a first time range that is longer than a second time range of a corresponding short-term demand predictions.
3. The non-transitory computer readable medium of claim 2 , wherein the at least one prediction model corresponds to a short-term prediction and a long-term prediction based on learning from input signals derived from observed behaviors of a multi-node computing system.
4. The non-transitory computer readable medium of claim 1 , wherein the acts further comprise forming a candidate set of non-critical microservices by accessing the microservices platform to identify a set of microservices hosted on the microservices platform, and eliminating microservices that are associated with a critical label from the candidate set.
5. The non-transitory computer readable medium of claim 4 , wherein the microservice is marked as critical based on monitoring of health metrics of the microservice during handling of a failure event.
6. The non-transitory computer readable medium of claim 1 , wherein the acts further comprise throttling a non-critical microservice when either the short-term demand prediction of the non-critical microservice is high or when the long-term demand prediction of the non-critical microservice is high.
7. The non-transitory computer readable medium of claim 6 , wherein the acts further comprise throttling additional non-critical microservices until the microservices platform is operating below a choke point threshold.
8. The non-transitory computer readable medium of claim 1 , wherein input signals derived from observed behaviors comprise at least one of, CPU demands, network services demands, or storage facilities demands.
9. A method comprising:
maintaining a microservices platform that responds to one or more throttling commands, wherein multiple throttling strategies are individually selectable by the microservices platform, individual ones of the multiple throttling strategies use at least one prediction model that outputs long-term and short-term demand predictions;
in response to receiving a first microservices throttling command at a first time under first conditions, using the at least one prediction model to select a first throttling strategy of the multiple throttling strategies to use to throttle a microservice, the first throttling strategy being selected based upon a first long-term or short-term demand prediction; and
in response to receiving a second microservices throttling command at a second time under second conditions, using the at least one prediction model to select a second throttling strategy of the multiple throttling strategies to use to throttle the microservice, the second throttling strategy being selected based upon a second long-term or short-term demand prediction,
wherein the first and second conditions are different, the first and second long-term or short-term demand predictions are different, and the first and second strategies are different.
10. The method of claim 9 , wherein respective long-term demand predictions correspond to a first time range that is longer than a second time range of a corresponding short-term demand predictions.
11. The method of claim 10 , wherein the at least one prediction model corresponds to a short-term prediction and a long-term prediction based on learning from input signals derived from observed behaviors of a multi-node computing system.
12. The method of claim 9 , further comprising forming a candidate set of non-critical microservices by accessing the microservices platform to identify a set of microservices hosted on the microservices platform, and eliminating microservices that are associated with a critical label from the candidate set.
13. The method of claim 12 , wherein the microservice is marked as critical based on monitoring of health metrics of the microservice during handling of a failure event.
14. The method of claim 9 , further comprising throttling a non-critical microservice when either the short-term demand prediction of the non-critical microservice is high or when the long-term demand prediction of the non-critical microservice is high.
15. The method of claim 14 , further comprising throttling additional non-critical microservices until the microservices platform is operating below a choke point threshold.
16. The method of claim 9 , wherein input signals derived from observed behaviors comprise at least one of, CPU demands, network services demands, or storage facilities demands.
17. A system comprising:
a storage medium having stored thereon a sequence of instructions; and
a processor that executes the sequence of instructions to cause acts comprising,
maintaining a microservices platform that responds to one or more throttling commands, wherein multiple throttling strategies are individually selectable by the microservices platform, individual ones of the multiple throttling strategies use at least one prediction model that outputs long-term and short-term demand predictions;
in response to receiving a first microservices throttling command at a first time under first conditions, using the at least one prediction model to select a first throttling strategy of the multiple throttling strategies to use to throttle a microservice, the first throttling strategy being selected based upon a first long-term or short-term demand prediction; and
in response to receiving a second microservices throttling command at a second time under second conditions, using the at least one prediction model to select a second throttling strategy of the multiple throttling strategies to use to throttle the microservice, the second throttling strategy being selected based upon a second long-term or short-term demand prediction,
wherein the first and second conditions are different, the first and second long-term or short-term demand predictions are different, and the first and second strategies are different.
18. The system of claim 17 , wherein respective long-term demand predictions correspond to a first time range that is longer than a second time range of a corresponding short-term demand predictions.
19. The system of claim 18 , wherein the at least one prediction model corresponds to a short-term prediction and a long-term prediction based on learning from input signals derived from observed behaviors of a multi-node computing system.
20. The system of claim 17 , wherein the acts further comprise forming a candidate set of non-critical microservices by accessing the microservices platform to identify a set of microservices hosted on the microservices platform, then eliminating microservices that are associated with a critical label from the candidate set.
21. The system of claim 20 , wherein the microservice is marked as critical based on monitoring of health metrics of the microservice during handling of a failure event.
22. The system of claim 17 , wherein the acts further comprise throttling a further non-critical microservice when either the short-term demand prediction of the further non-critical microservice is high or when the long-term demand prediction of the further non-critical microservice is high.
23. The system of claim 22 , wherein the acts further comprise throttling additional non-critical microservices until the microservices platform is operating below a choke point threshold.
24. The system of claim 17 , wherein input signals derived from observed behaviors comprise at least one of, CPU demands, network services demands, or storage facilities demands.