Adversarial automated reinforcement-learning-based application-manager training
The current document is directed to automated reinforcement-learning-based application managers that that are trained using adversarial training. During adversarial training, potentially disadvantageous next actions are selected for issuance by an automated reinforcement-learning-based application manager at a lower frequency than selection of next actions, according to a policy that is learned to provide optimal or near-optimal control over a computing environment that includes one or more applications controlled by the automated reinforcement-learning-based application manager. By selecting disadvantageous actions, the automated reinforcement-learning-based application manager is forced to explore a much larger subset of the system-state space during training, so that, upon completion of training, the automated reinforcement-learning-based application manager has learned a more robust and complete optimal or near-optimal control policy than had the automated reinforcement-learning-based application manager been trained by simulators or using management actions and computing-environment responses recorded during previous controlled operation of a computing-environment.
1. An automated reinforcement-learning-based application manager that manages a computing environment that includes one or more applications and one or more of a distributed computing system having multiple computer systems interconnected by one or more networks, a standalone computer system, and a processor-controlled user device, the reinforcement-learning based application manager comprising:
one or more processors, one or more memories, and one or more communications subsystems;
a set of actions A that can be issued to the computing environment;
an iterative control process that repeatedly
when adversarial training is not occurring,
selects and issues a next action to the computing environment according to a positive control policy that uses a state vector that represents a current state of the computational environment,
when adversarial training is occurring,
selects and issues, at a first frequency, a next action to the computing environment according to the positive control policy, and
selects and issues at a second frequency less than the first frequency, a next action to the computing environment according to a negative control policy, and
receives, from the computing environment, a next state and a reward, which the control process uses to attempt to learn an optimal or near-optimal control policy.
2. The automated reinforcement-learning-based application manager of claim 1 further including a second set of actions B from which the negative control policy selects a next action.
3. The automated reinforcement-learning-based application manager of claim 2 wherein the negative control policy selects actions from either the set of actions B or from the set of actions A.
4. The automated reinforcement-learning-based application manager of claim 1 wherein, when adversarial training is occurring and a next action a′ is selected according to the negative control policy, actions complementary to the next action a′ are temporarily removed from the set of actions A so that the automated reinforcement-learning-based application manager cannot immediately reverse the effects of action a′ in subsequent iterative-control-process cycles.
5. The automated reinforcement-learning-based application manager of claim 1 wherein the positive control policy attempts to select a next action that causes a transition to a next state with a maximum possible value.
6. The automated reinforcement-learning-based application manager of claim 1 wherein the positive control policy attempts to select a next action that causes a transition to a next state most likely to result in a maximum cumulative reward over subsequent iterative-control-process cycles.
7. The automated reinforcement-learning-based application manager of claim 1 wherein the negative control policy attempts to select a next action that causes a transition to a next state with a minimum possible value.
8. The automated reinforcement-learning-based application manager of claim 1 wherein the negative control policy attempts to select a next action that causes a transition to a next state most likely to result in a minimum cumulative reward over subsequent iterative-control-process cycles.
9. The automated reinforcement-learning-based application manager of claim 1 wherein, during adversarial training, the automated reinforcement-learning-based application manager includes two iterative control processes, one that uses the positive control policy and one that uses the negative control policy.
10. A method that trains an automated reinforcement-learning-based application manager that manages a computing environment that includes one or more applications and one or more of a distributed computing environment having multiple computer systems interconnected by one or more networks, a standalone computer system, and a processor-controlled user device, the automated reinforcement-learning-based application manager having one or more processors, one or more memories, one or more communications subsystems, and a set of actions A that can be issued to the computing environment, the method comprising:
iteratively, by an iterative control process,
when adversarial training is not occurring,
selecting and issuing a next action to the computing environment according to a positive control policy that uses a state vector that represents a current state of the computational environment,
when adversarial training is occurring,
selecting and issuing, at a first frequency, a next action to the computing environment according to the positive control policy, and
selecting and issuing at a second frequency less than the first frequency, a next action to the computing environment according to a negative control policy, and
receiving, from the computing environment, a next state and a reward, which the automated reinforcement-learning-based application manager uses to attempt to learn an optimal or near-optimal control policy.
11. The method of claim 10 further including a second set of actions B from which the negative control policy selects a next action.
12. The method of claim 11 wherein the negative control policy selects actions from either the set of actions B or from the set of actions A.
13. The method of claim 10 wherein, when adversarial training is occurring and a next action a′ is selected according to the negative control policy, actions complementary to the next action a′ are temporarily removed from the set of actions A so that the automated reinforcement-learning-based application manager cannot immediately reverse the effects of action a′ in subsequent iterative-control-process cycles.
14. The method of claim 10 wherein the positive control policy attempts to select a next action that causes a transition to a next state with a maximum possible value.
15. The method of claim 10 wherein the positive control policy attempts to select a next action that causes a transition to a next state most likely to result in a maximum cumulative reward over subsequent iterative-control-process cycles.
16. The method of claim 10 wherein the negative control policy attempts to select a next action that causes a transition to a next state with a minimum possible value.
17. The method of claim 10 wherein the negative control policy attempts to select a next action that causes a transition to a next state most likely to result in a minimum cumulative reward over subsequent iterative-control-process cycles.
18. The method of claim 10 wherein, during adversarial training, the automated reinforcement-learning-based application manager includes two iterative control processes, one that uses the positive control policy and one that uses the negative control policy.
19. A physical data-storage device encoded with computer instructions that, when executed by one or more processors of a computer system that implements an automated reinforcement-learning-based application manager having one or more processors, one or more memories, one or more communications subsystems, a set of actions A that can be issued to a computing environment, controls the automated reinforcement-learning-based application manager to:
iteratively, by an iterative control process,
when adversarial training is not occurring,
selecting and issuing a next action to the computing environment according to a positive control policy that uses a state vector that represents a current state of the computational environment,
when adversarial training is occurring,
selecting and issuing, at a first frequency, a next action to the computing environment according to the positive control policy, and
selecting and issuing at a second frequency less than the first frequency, a next action to the computing environment according to a negative control policy, and
receiving, from the computing environment, a next state and a reward, which the automated reinforcement-learning-based application manager uses to attempt to learn an optimal or near-optimal control policy.
20. The physical data-storage of claim 19 wherein, when adversarial training is occurring and a next action a′ is selected according to the negative control policy, actions complementary to the next action a′ are temporarily removed from the set of actions A so that the automated reinforcement-learning-based application manager cannot immediately reverse the effects of action a′ in subsequent iterative-control-process cycles.