Reward function generation method and computer system

1. A reward function generation method for calculating a reward in reinforcement learning, the reward function generation method being executed by a computer,

the computer including a processor, a memory connected to the processor, and a network interface connected to the processor,

the reward function generation method comprising:

a first step of accepting input of an instruction to generate a reward function, by the processor, wherein the instruction to generate the reward function includes:

a plurality of setting data comprising information regarding a key performance indicator, a control purpose of the key performance indicator, and a timing of calculating a reward regarding the key performance indicator, wherein the memory stores function definition information in which a combination of the control purpose of the key performance indicator and the timing of calculating the reward are associated with a type of a specific function, and

definition information on the key performance indicator;

a second step of generating one partial reward function for one of the plurality of setting data and storing the one partial reward function in the memory, by the processor, wherein the second step includes:

selecting target setting data from the plurality of setting data,

referring to the function definition information based on the combination of the control purpose of the key performance indicator and the timing of calculating the reward included in the target setting data, and identifying the type of the specific function to be generated, and

generating the specific function having a parameter related to the key performance indicator as a variable based on the identified type of the specific function and the definition information on the key performance indicator; and

a third step of generating a linear combination of a plurality of partial reward functions as the reward function and storing the linear combination in the memory, by the processor; and

a fourth step of outputting information regarding the generated reward function to a computer that executes the reinforcement learning, by the processor.

2. The reward function generation method according to claim 1 , wherein

the plurality of setting data further comprises a degree of priority indicating order of priority of the control purpose of the key performance indicator, and

the second step further includes:

calculating a scale factor that defines a magnitude of a value calculated by using the specific function based on a degree of priority included in the target setting data, by the processor;

calculating a weight based on a domain of the key performance indicator included in the target setting data, by the processor; and

generating the one partial reward function based on the specific function, the scale factor, and the weight, by the processor.

3. The reward function generation method according to claim 2 , further comprising:

a fifth step of acquiring a result of the reinforcement learning using the generated reward function, and generating an evaluation result by evaluating the result of the reinforcement learning, by the processor;

a sixth step of identifying a partial reward function to be corrected based on the evaluation result, and correcting the identified partial reward function, by the processor; and

a seventh step of updating the reward function based on a correction result of the identified partial reward function, by the processor.

4. The reward function generation method according to claim 3 , wherein

the control purpose of the key performance indicator includes a constraint to be satisfied by the key performance indicator,

the reward function includes a first partial reward function generated based on the plurality of setting data including a first constraint as the control purpose of the key performance indicator, and

the sixth step includes identifying the first partial reward function as the partial reward function to be corrected, by the processor, when the first constraint is not satisfied.

5. The reward function generation method according to claim 3 , wherein

the one partial reward function is managed in association with the plurality of setting data,

the fifth step includes calculating a degree of contribution indicating a magnitude of contribution of a value obtained from each of the plurality of partial reward functions to selection of an action in the reinforcement learning, by the processor, and

the sixth step includes:

comparing a first sort order that is an order in which the plurality of partial reward functions are sorted based on a degree of priority included in the plurality of setting data corresponding to the one partial reward function with a second sort order that is an order in which the partial reward functions are sorted based on the degree of contribution, by the processor; and

identifying a partial reward function of which the first sort order and the second sort order are different from each other, as the partial reward function to be corrected, by the processor.

6. The reward function generation method according to claim 3 , further comprising

providing an interface for setting the plurality of setting data and a correction method of the partial reward function to be corrected, by the processor.

7. A computer system comprising a plurality of computers,

the plurality of computers each including a processor, a memory connected to the processor, and a network interface connected to the processor,

the computer system including a learning unit that executes reinforcement learning, and a reward function generation unit that generates a reward function for calculating a reward in the reinforcement learning, wherein

the reward function generation unit,

generates one partial reward function for one of the plurality of setting data when accepting input of an instruction to generate the reward function, wherein the instruction to generate the reward function includes:

a plurality of setting data comprising information regarding a key performance indicator, a control purpose of the key performance indicator, and a timing of calculating a reward regarding the key performance indicator, and

definition information on the key performance indicator;

manages function definition information in which a combination of the control purpose of the key performance indicator and the timing of calculating the reward are associated with a type of a specific function;

generates a linear combination of a plurality of partial reward functions as the reward function;

selects target setting data from the plurality of setting data included in the instruction to generate the reward function;

refers to the function definition information based on the combination of the control purpose of the key performance indicator and the timing of calculating the reward included in the target setting data, and identifies the type of the specific function to be generated;

generates a specific function having a parameter related to the key performance indicator as a variable based on the identified type of the specific function and the definition information on the key performance indicator; and

outputs, to the learning unit, information regarding the generated reward function.

8. The computer system according to claim 7 , wherein

the plurality of setting data further comprises a degree of priority indicating order of priority of the control purpose of the key performance indicator, and

the reward function generation unit

calculates a scale factor that defines a magnitude of a value calculated by using the specific function based on a degree of priority included in the target setting data,

calculates a weight based on a domain of the key performance indicator included in the target setting data, and

generates the one partial reward function based on the specific function, the scale factor, and the weight.

9. The computer system according to claim 8 , wherein

the learning unit

executes the reinforcement learning by using the reward function generated, and

transmits, to the reward function generation unit, a result of the reinforcement learning by using the reward function generated, and

the reward function generation unit

evaluates the result of the reinforcement learning by using the reward function generated,

identifies a partial reward function to be corrected based on a result of the evaluation, and corrects the identified partial reward function, and

updates the reward function based on the basis of a correction result of the identified partial reward function.

10. The computer system according to claim 9 , wherein

the control purpose of the key performance indicator includes a constraint to be satisfied by the key performance indicator,

the reward function includes a first partial reward function generated based on the plurality of setting data including a first constraint as the control purpose of the key performance indicator, and

the reward function generation unit identifies the first partial reward function as the partial reward function to be corrected when the first constraint is not satisfied.

11. The computer system according to claim 9 , wherein

the one partial reward function is managed in association with the plurality of setting data, and

the reward function generation unit

calculates a degree of contribution indicating a magnitude of contribution of a value obtained from each of the plurality of partial reward functions to selection of an action in the reinforcement learning,

compares a first sort order that is an order in which the plurality of partial reward functions are sorted based on a degree of priority included in the plurality of setting data corresponding to the one partial reward function with a second sort order that is an order in which the plurality of partial reward functions are sorted based on the degree of contribution, and

identifies a partial reward function of which the first sort order and the second sort order are different from each other, as the partial reward function to be corrected.

12. The computer system according to claim 9 , wherein

the reward function generation unit provides an interface for setting the plurality of setting data and a correction method of the partial reward function to be corrected.