Transferable training for automated reinforcement-learning-based application-managers
The current document is directed to transfer of training received by a first automated reinforcement-learning-based application manager while controlling a first application is transferred to a second automated reinforcement-learning-based application manager which controls a second application different from the first application. Transferable training provides a basis for automated generation of applications from application components. Transferable training is obtained from composition of applications from application components and composition of reinforcement-learning-based-control-and-learning constructs from reinforcement-learning-based-control-and-learning constructs of application components.
1. An automated reinforcement-learning-based application manager that manages a computing environment that includes one or more applications and one or more of a distributed computing system having multiple computer systems interconnected by one or more networks, a standalone computer system, and a processor-controlled user device, the reinforcement-learning based application manager comprising:
one or more processors, one or more memories, and one or more communications subsystems;
a set of actions A that can be issued to the computing environment; and
an iterative control process that repeatedly
when initial training is not occurring,
selects and issues a next action to the computing environment according to a control policy that uses a state vector that represents a current state of the computational environment,
when initial training is occurring,
selects and issues a next action to the computing environment according to a training control policy that uses a state vector that represents a current state of the computational environment and training information incorporated into the automated reinforcement-learning-based application manager that was acquired by a different automated reinforcement-learning-based application manager, and
receives, from the computing environment, a next state and a reward, which the control process uses to attempt to learn an optimal or near-optimal control policy.
2. The automated reinforcement-learning-based application manager of claim 1 wherein the training control policy uses a state vector that represents a current state of the computing environment and training information incorporated into the automated reinforcement-learning-based application manager to select a next action by:
generating a hidden-state vector for each of multiple components of the computing environment;
applying, to each hidden-state vector, a component-associated control policy for the component of the computing environment for which the hidden-state vector was generated to select an action; and
combining one or more of the actions selected by the component-associated control policies to produce the next action.
3. The automated reinforcement-learning-based application manager of claim 2 wherein the component-associated control policies include:
component-associated control policies associated with components for which training data for related components has been incorporated into the automated reinforcement-learning-based application manager;
static deterministic or stochastic component-associated control policies associated with components for which training data has been incorporated into the automated reinforcement-learning-based application manager; and
static deterministic or stochastic component-associated control policies associated with components comprising subcomponents for which training data has been incorporated into the automated reinforcement-learning-based application manager.
4. The automated reinforcement-learning-based application manager of claim 3 wherein the component-associated control policies associated with components for which training data for related components has been incorporated into the automated reinforcement-learning-based application manager employing exploratory action selection from an action set corresponding to the component.
5. The automated reinforcement-learning-based application manager of claim 2 wherein generating a hidden-state vector for each of multiple components of the computing environment further comprises:
decomposing the computing environment into components;
decomposing the state vector into component subvectors, each component subvector corresponding to a computing-environment component; and
applying a hidden-state-vector function to each component subvector to generate the hidden-state vector.
6. The automated reinforcement-learning-based application manager of claim 1 wherein initial training is discontinued after the automated reinforcement-learning-based application manager has learned a near-optimal or optimal control policy for the computing environment.
7. A method for transferring training data from one or more trained automated reinforcement-learning-based application managers to a target automated reinforcement-learning-based application manager that manages a computing environment that includes one or more applications and one or more of a distributed computing environment having multiple computer systems interconnected by one or more networks, a standalone computer system, and a processor-controlled user device, the automated reinforcement-learning-based application manager having one or more processors, one or more memories, one or more communications subsystems, and a set of actions A that can be issued to the computing environment, the method comprising:
decomposing the computing into components;
identifying training data for each of the components;
incorporating the identified training data into the target automated reinforcement-learning-based application manager; and
iteratively, by an iterative control process,
selecting and issuing a next action to the computing environment according to a control policy that uses a state vector that represents a current state of the computational environment and the training information incorporated into the automated reinforcement-learning-based application manager, and
receiving, from the computing environment, a next state and a reward, which the control process uses to attempt to learn an optimal or near-optimal control policy.
8. The method of claim 7 wherein the control policy comprises multiple component-associated control policies, each component-associated control policy selecting actions from a set of actions issuable to the component associated with the component-associated control policy.
9. The method of claim 8 wherein selecting and issuing a next action further comprises:
decomposing the state vector into subvectors, each subvector corresponding to one of the components;
generating a hidden-state vector from each state vector;
applying, to each hidden-state vector, a component-associated control policy; and
combining one or more of the actions selected by the component-associated control policies to produce the next action.
10. The method of claim 9 wherein the component-associated control policies include:
component-associated control policies associated with components for which training data for related components has been incorporated into the automated reinforcement-learning-based application manager;
static deterministic or stochastic component-associated control policies associated with components for which training data has been incorporated into the automated reinforcement-learning-based application manager; and
static deterministic or stochastic component-associated control policies associated with components comprising subcomponents for which training data has been incorporated into the automated reinforcement-learning-based application manager.
11. The method of claim 9 wherein the reward is computed by a functional composition of reward functions for each of the components.
12. The method of claim 9 wherein the training data comprises one or more of state-value functions and state/action-value functions.
13. A method that generates a new application for management by a target automated reinforcement-learning-based application manager that manages a computing environment that includes the new application and one or more of a distributed computing environment having multiple computer systems interconnected by one or more networks, a standalone computer system, and a processor-controlled user device, the automated reinforcement-learning-based application manager having one or more processors, one or more memories, one or more communications subsystems, and a set of actions A that can be issued to the computing environment, the method comprising:
carrying out an optimization method to select, based on an application specification, stored component information, and a set of constraints, a set of components from which the application is assembled;
incorporating the new application into the computing environment;
for each component associated with training data obtained during control of the component by an automated reinforcement-learning-based application manager, incorporating the identified training data into the target automated reinforcement-learning-based application manager; and
launching operation of the target automated reinforcement-learning-based application manager.
14. The method of claim 13 wherein the set of constraints includes:
maximizing the number of components associated with training data; and
minimizing the number of components.
15. The method of claim 13 wherein the target automated reinforcement-learning-based application manager
iteratively, by an iterative control process,
selects and issues a next action to the computing environment according to a control policy that uses a state vector that represents a current state of the computational environment and the training information incorporated into the automated reinforcement-learning-based application manager, and
receives, from the computing environment, a next state and a reward, which the control process uses to attempt to learn an optimal or near-optimal control policy.
16. The method of claim 15 wherein the control policy comprises multiple component-associated control policies, each component-associated control policy selecting actions from a set of actions issuable to the component associated with the component-associated control policy.
17. The method of claim 16 wherein selecting and issuing a next action further comprises:
decomposing the state vector into subvectors, each subvector corresponding to one of the components;
generating a hidden-state vector from each state vector;
applying, to each hidden-state vector, a component-associated control policy; and
combining one or more of the actions selected by the component-associated control policies to produce the next action.
18. The method of claim 17 wherein the component-associated control policies include:
component-associated control policies associated with components for which training data for related components has been incorporated into the automated reinforcement-learning-based application manager;
static deterministic or stochastic component-associated control policies associated with components for which training data has been incorporated into the automated reinforcement-learning-based application manager; and
static deterministic or stochastic component-associated control policies associated with components comprising subcomponents for which training data has been incorporated into the automated reinforcement-learning-based application manager.
19. The method of claim 17 wherein the reward is computed by a functional composition of reward functions for each of the components.
20. The method of claim 17 wherein the training data comprises one or more of state-value functions and state/action-value functions.
21. A physical data-storage device encoded with computer instructions that, when executed by one or more processors of a computer system that implements an automated reinforcement-learning-based application manager having one or more processors, one or more memories, one or more communications subsystems, a set of actions A that can be issued to a computing environment, controls the automated reinforcement-learning-based application manager to:
iteratively, by an iterative control process,
selecting and issuing a next action to the computing environment according to a control policy that uses a state vector that represents a current state of the computational environment and the training information incorporated into the automated reinforcement-learning-based application manager, and
receiving, from the computing environment, a next state and a reward, which the control process uses to attempt to learn an optimal or near-optimal control policy.