Modular reinforcement-learning-based application manager

1. A modular reinforcement-learning-based application manager that manages one or more applications and a computing environment, within which the applications run, comprising one or more of a distributed computing system having multiple computer systems interconnected by one or more networks, a standalone computer system, and a processor-controlled user device, the modular reinforcement-learning based application manager comprising:

a reinforcement-learning-based application manager that receives rewards and observations from the computing environment and issues actions to the computing environment in accordance with an internally maintained policy; and

an interface between the computing environment and the reinforcement-learning-based application manager that

transforms a manager-specific action issued by the reinforcement-learning-based application manager to a generic action and issues the generic action to the computing environment;

provides, to the computing environment, generic-actions metadata for the generic actions issued to the computing environment;

transforms a generic observation generated by the computing environment to an equivalent manager-specific observation and issues the manager-specific observation to the reinforcement-learning-based application manager; and

provides generic-observations metadata.

2. The modular reinforcement-learning-based application manager of claim 1

wherein the manager-specific action is encoded as one or more numerical values;

wherein the generic action is encoded, by the interface, in a generic-action data structure that includes data generated from the numerical values in the manager-specific action; and

wherein the data in the generic-action data structure is described by the generic-actions metadata.

3. The modular reinforcement-learning-based application manager of claim 2 wherein the computing environment:

uses the generic-actions metadata to transform the generic action into one or more commands that implement the generic action; and

inputs the one or more commands to components and/or subsystems in the computing environment to carry out the generic action.

4. The modular reinforcement-learning-based application manager of claim 1

wherein the manager-specific observation is encoded as one or more numerical values;

wherein the generic observation is encoded, by the computing environment, in a generic-observation data structure; and

wherein the data in the generic-observation data structure is described by the generic-observations metadata.

5. The modular reinforcement-learning-based application manager of claim 4 wherein the computing environment:

uses the generic-observations metadata to determine information sources within the computing environment, and access methods to request information from the information sources, to obtain information needed to generate the generic observation;

uses the generic-observations metadata to encode the obtained information in the generic-observation data structure; and

inputs the generic-observation data structure to the interface.

6. The modular reinforcement-learning-based application manager of claim 1 wherein the generic-observations metadata includes:

data that indicates the number of elements in a generic-observation;

a description of each element in a generic-observation; and

for each data component of a generic-observation element,

an indication of the data type for the data, and

an indication of what the data component represents.

7. The modular reinforcement-learning-based application manager of claim 1 wherein the generic-actions metadata includes:

data that indicates the number of different generic actions; and

for each different generic action,

an indication of the number of data components for the generic action, and

for each data component,

an indication of a data type, and

an indication of what the data component represents.

8. The modular reinforcement-learning-based application manager of claim 1 wherein the computing environment additionally includes a reward module that generates user-specified rewards that are issued to the reinforcement-learning-based application manager.

9. The modular reinforcement-learning-based application manager of claim 8 wherein the reward module provides a user interface through which a user defines a reward function that receives observation data and action data and outputs a numeric reward.

10. The modular reinforcement-learning-based application manager of claim 1 wherein the reinforcement-learning-based application manager maintains a policy and a belief distribution, and updates the policy, during training, to generate, over time, a near-optimal or optimal policy, where a near-optimal policy is closer to an optimal policy than the policies achievable by human administrators and managers.

11. The modular reinforcement-learning-based application manager of claim 10 wherein the policy returns a next action to issue to the computing environment from the current belief distribution.

12. A method for interfacing a computing environment to a reinforcement-learning-based application manager, the method comprising:

incorporating, by the computing environment comprising one or more of a distributed computing system having multiple computer systems interconnected by one or more networks, a standalone computer system, and a processor-controlled user device, an interface that interfaces the computing environment to the reinforcement-learning-based application manager;

transforming, by the interface, a manager-specific actions issued by the reinforcement-learning-based application manager to corresponding generic actions and issuing the generic actions to the computing environment;

providing to the computing environment, by the interface, generic-actions metadata for the generic actions issued to the computing environment;

transforming, by the interface, generic observations generated by the computing environment to equivalent manager-specific observations and issuing, by the interface, the manager-specific observations to the reinforcement-learning-based application manager; and

providing, by the interface, generic-observations metadata.

13. The method of claim 12

wherein the manager-specific actions are each encoded as one or more numerical values;

wherein the generic actions are each encoded, by the interface, in a generic-action data structure that includes data generated from the numerical values in the manager-specific action; and

wherein the data in each generic-action data structure is described by the generic-actions metadata.

14. The method of claim 13 further comprising:

using, by the computing environment, the generic-actions metadata to transform each generic action into one or more commands that implement the generic action; and

inputting, by the computing environment, the one or more commands corresponding to each generic action to components and/or subsystems in the computing environment to carry out the generic action.

15. The method of claim 12

wherein the manager-specific observations are each encoded as one or more numerical values;

wherein the generic observations are each encoded, by the computing environment, in a generic-observation data structure; and

wherein the data in each generic-observation data structure is described by the generic-observations metadata.

16. The method of claim 15 further including:

using, by the computing environment, the generic-observations metadata to determine information sources within the computing environment, and access methods to request information from the information sources, to obtain information needed to generate each generic observation;

using, by the computing environment, the generic-observations metadata to encode the obtained information for each generic observation, in a generic-observation data structure; and

inputting the generic-observation data structure for each generic observation to the interface.

17. A physical data-storage device encoded with computer instructions that implement an interface between a computing environment and a reinforcement-learning-based application manager, the computing environment comprising one or more of a distributed computer system having multiple computer systems interconnected by one or more networks, a standalone computer system, and a processor-controlled user device, and the interface

transforming a manager-specific actions issued by the reinforcement-learning-based application manager to corresponding generic actions and issuing the generic actions to the computing environment;

providing, to the computing environment, generic-actions metadata for the generic actions issued to the computing environment;

transforming generic observations generated by the computing environment to equivalent manager-specific observations and issuing the manager-specific observations to the reinforcement-learning-based application manager; and

providing generic-observations metadata for the generic observations.

18. The physical data-storage device of claim 17

wherein the manager-specific actions are each encoded as one or more numerical values;

wherein the generic actions are each encoded, by the interface, in a generic-action data structure that includes data generated from the numerical values in the manager-specific action; and

wherein the data in each generic-action data structure is described by the generic-actions metadata.

19. The physical data-storage device of claim 18

wherein the computing environment uses the generic-actions metadata to transform each generic action into one or more commands that implement the generic action; and

wherein the computing environment uses the one or more commands corresponding to each generic action to components and/or subsystems in the computing environment to carry out the generic action.

20. The physical data-storage device of claim 17

wherein the manager-specific observations are each encoded as one or more numerical values;

wherein the generic observations are each encoded, by the computing environment, in a generic-observation data structure; and

wherein the data in each generic-observation data structure is described by the generic-observations metadata.

21. The physical data-storage device of claim 20

wherein the computing environment uses the generic-observations metadata to determine information sources within the computing environment, and access methods to request information from the information sources, to obtain information needed to generate each generic observation;

wherein the computing environment uses the generic-observations metadata to encode the obtained information for each generic observation, in a generic-observation data structure; and

wherein the generic-observation inputs a data structure for each generic observation to the interface.