Reinforcement learning policy serving and training framework in production cloud systems
Method and systems for online training management of reinforcement learning policy serving for cloud computing systems are discloses. An example method includes controlling a cloud computing system using a first reinforcement learning (RL) model; training the first RL model to generate a second RL model in response to one or more first criteria being satisfied; and controlling the cloud computing system using the second RL model in response to one or more second criteria being satisfied.
1 . A method, comprising:
controlling resources of a cloud computing system using a first reinforcement learning (RL) model, the first RL model performs tasks to automate control of the resources while refraining from performing online training of the first RL model, wherein the first RL model, for each of the tasks, receives state information from the cloud computing system, returns an action to the cloud computing system, receives reward information from the cloud computing system, and records a first trajectory comprising the state, action, and reward for each of the tasks;
monitoring at least one first trajectory generated while using the first RL model to identify a first trigger to start online training of the first RL model;
performing online training of the first RL model to generate a second RL model in response to the first trigger;
monitoring one or more second trajectories generated while performing the online training of the first RL model, to identify a second trigger to stop the online training of the first RL model; and
controlling the resources of the cloud computing system using the second RL model in response to the second trigger while refraining from performing online training of the second RL model.
2 . The method of claim 1 , wherein monitoring the at least one first trajectory further comprises evaluating, based on the at least one first trajectory, one or more first properties of the reward information received from the cloud computing system for each of the tasks, the one or more first properties of the reward information comprises one or more performance metrics or utilization metrics associated with the action for each task; and
wherein monitoring the one or more second trajectories further comprises evaluating, based on the one or more second trajectories, one or more second properties of the reward information received from the cloud computing system for each of the tasks, the one or more second properties of the reward information comprises one or more performance metrics or utilization metrics associated with the action for each task.
3 . The method of claim 2 , wherein the first trigger to start online training of the first RL model is identified based on the one or more performance metrics or utilization metrics of the one or more first properties of the reward information being less than a first threshold; and
wherein the second trigger to stop online training of the first RL model is identified based on the one or more performance metrics or utilization metrics of the one or more second properties of the reward information being less than or equal to a second threshold or being greater than or equal to a third threshold.
4 . The method of claim 3 , wherein:
the one or more performance metrics or utilization metrics of the one or more first properties of the reward information include an average reward value among a number of past trajectories generated using the first RL model and wherein the one or more performance metrics or utilization metrics of the one or more second properties of the reward information include a variance of reward values among a number of past trajectories generated using the first RL model during the online training or an average reward value among the number of past trajectories generated using the first RL model during the online training.
5 . The method of claim 2 , further comprising:
storing the at least one first trajectory and the one or more second trajectories in a database;
wherein evaluating the one or more first properties comprises evaluating the one or more first properties via the database; and
wherein evaluating the one or more second properties comprises evaluating the one or more second properties via the database.
6 . The method of claim 1 , further comprising:
outputting a first indication to start the online training the first RL model in response to the first trigger; and
outputting a second indication to stop the online training the first RL model in response to the second trigger.
7 . The method of claim 1 , wherein controlling resources of the cloud computing system comprises:
controlling resource autoscaling associated with the resources of the cloud computing system;
controlling power management associated with the resources of the cloud computing system;
controlling load balancing associated with the resources of the cloud computing system;
controlling congestion control associated with the resources of the cloud computing system;
controlling database query performance associated with the resources of the cloud computing system; or
a combination thereof.
8 . The method of claim 1 , wherein performing the online training of the first RL model comprises performing online training of the first RL model while using the first RL model to perform the tasks to automate control of the resources.
9 . A system, comprising:
a memory; and
one or more processors coupled to the memory, the one or more processors being configured to:
control resources of a cloud computing system using a first reinforcement learning (RL) model, the first RL model performs tasks to automate control of the resources while refraining from performing online training of the first RL model, wherein the first RL model, for each of the tasks, receives state information from the cloud computing system, returns an action to the cloud computing system, receives reward information from the cloud computing system, and records a first trajectory comprising the state, action, and reward for each of the tasks;
monitor at least one first trajectory generated while using the first RL model, to identify a first trigger to start online training of the first RL model;
perform online training of the first RL model to generate a second RL model in response to the first trigger;
monitor one or more second trajectories generated while performing the online training of the first RL model, to identify a second trigger to stop the online training of the first RL model; and
control the resources of the cloud computing system using the second RL model in response to the second trigger, while refraining from performing online training of the second RL model.
10 . The system of claim 9 , wherein the one or more processors are further configured to:
evaluate, based on the at least one first trajectory, one or more first properties of the reward information received from the cloud computing system for each of the tasks, the one or more first properties of the reward information comprises one or more performance metrics or utilization metrics associated with the action for each task; and
evaluate, based on the one or more second trajectories, one or more second properties of the reward information received from the cloud computing system for each of the tasks, the one or more second properties of the reward information comprises one or more performance metrics or utilization metrics associated with the action for each task.
11 . The system of claim 10 , wherein the first trigger to start online training of the first RL model is identified based on the one or more performance metrics or utilization metrics of the one or more first properties of the reward information
being less than a first threshold; and
wherein the second trigger to stop online training of the first RL model is identified based on the one or more performance metrics or utilization metrics of the one or more second properties of the reward information being less than or equal to a second threshold, or being greater than or equal to a third threshold.
12 . The system of claim 11 , wherein:
the one or more performance metrics or utilization metrics of the one or more first properties of the reward information include an average reward value among a number of past trajectories generated using the first RL model; wherein the one or more performance metrics or utilization metrics of the one or more second properties of the reward information include a variance of reward values among a number of past trajectories generated using the first RL model during the online training; or an average reward value among the number of past trajectories generated using the first RL model during the online training.
13 . The system of claim 10 , wherein:
the one or more processors are further configured to store the at least one first trajectory and the one or more second trajectories in a database;
the one or more processors are further configured to evaluate the one or more first properties via the database; and
the one or more processors are further configured to evaluate the one or more second properties via the database.
14 . The system of claim 9 , wherein the one or more processors are further configured to:
output a first indication to start the online training the first RL model in response to the first trigger; and
output a second indication to stop the online training the first RL model in response to the second trigger.
15 . The system of claim 9 , wherein to control resources of the cloud computing system, the one or more processors are further configured to:
control resource autoscaling associated with the resources of the cloud computing system;
control power management associated with the resources of the cloud computing system;
control load balancing associated with the resources of the cloud computing system;
control congestion control associated with the resources of the cloud computing system;
control database query performance associated with the resources of the cloud computing system; or
a combination thereof.
16 . The system of claim 9 , wherein to perform online training of the first RL model, the one or more processors are further configured to perform the online training of the first RL model while using the first RL model to perform the tasks to automate control of the resources.
17 . A computer program product for online training management, the computer program product comprising:
a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to:
control resources of a cloud computing system using a first reinforcement learning (RL) model, the first RL model performs tasks to automate control of the resources while refraining from performing online training of the first RL model, wherein the first RL model, for each of the tasks, receives state information from the cloud computing system, returns an action to the cloud computing system, receives reward information from the cloud computing system, and records a first trajectory comprising the state, action, and reward for each of the tasks;
monitor at least one first trajectory generated while using the first RL model, to identify a first trigger to start online training of the first RL model;
perform online training of the first RL model to generate a second RL model in response to the first trigger;
monitor one or more second trajectories generated while performing the online training of the first RL model, to identify a second trigger to stop the online training of the first RL model; and
control the resources of the cloud computing system using the second RL model in response to the second trigger, while refraining from performing online training of the second RL model.
18 . The computer program product of claim 17 , wherein the computer-readable program code being further executable by the one or more computer processors to:
evaluate, based on the at least one first trajectory, one or more first properties of the reward information received from the cloud computing system for each of the tasks, the one or more first properties of the reward information comprises one or more performance metrics or utilization metrics associated with the action for each task; and
evaluate, based on the one or more second trajectories, one or more second properties of the reward information received from the cloud computing system for each of the tasks, the one or more second properties of the reward information comprises one or more performance metrics or utilization metrics associated with the action for each task.
19 . The computer program product of claim 18 , wherein the first trigger to start online training of the first RL model is identified based on the one or more performance metrics or utilization metrics of the one or more first properties of the reward information
being less than a first threshold; and
wherein the second trigger to stop online training of the first RL model is identified based on the one or more performance metrics or utilization metrics of the one or more second properties of the reward information being less than or equal to a second threshold, or being greater than or equal to a third threshold.
20 . The computer program product of claim 19 , wherein:
the one or more performance metrics or utilization metrics of the one or more first properties of the reward information include an average reward value among a number of past trajectories generated using the first RL model; wherein the one or more performance metrics or utilization metrics of the one or more second properties of the reward information include a variance of reward values among a number of past trajectories generated using the first RL model during the online training, or an average reward value among the number of past trajectories generated using the first RL model during the online training.