Learning device, learning method, control system, and recording medium
A learning device uses, on the basis of a second state corresponding to a first action of a control subject in a first state and a second action calculated from the second state using a policy model, uses a plurality of evaluation models that each calculate a second evaluation value, which is an index value of the goodness of the second action in the second state, to respectively calculate the second evaluation values; and updates the evaluation model on the basis of the smallest second evaluation value among the plurality of second evaluation values, and a first evaluation value, which is an index value of the goodness of the first action in the first state.
1 . A learning device comprising:
a memory configured to store instructions; and
one or more processors configured to execute the instructions to:
associate a priority based on a magnitude of an error with an experience, the experience is a combination of a first state of a control subject, a first action of the control subject in the first state, a reward according to the first action in the first state, and a second state according to the first action in the first state, wherein the error is between
1) an evaluation value obtained based on a minimum evaluation value among evaluation values for the second action in the second state, which is the evaluation value in the first state indicated in the experience, and which is the evaluation value for the first action indicated in the experience, the evaluation value being obtained by applying the reward indicated in the experience, the second state indicated in the experience, and the second action calculated from the second state using a policy model to a plurality of evaluation models obtained by setting a plurality of parameter values in an evaluation model body, and
2) an evaluation value obtained for each evaluation model by applying the first state indicated in the experience and the first action indicated in the experience to a plurality of evaluation models obtained by setting a plurality of parameter values in the evaluation model body, and is a maximum evaluation value among the evaluation values for the first action in the first state;
acquire the experience based on the priority;
apply the second state indicated in the experience acquired and the second action calculated from the second state using the policy model to the plurality of evaluation models obtained by setting the plurality of parameter values in the evaluation model body, and calculating the evaluation value for the second action in the second state for each evaluation model; and
update the evaluation model based on a minimum evaluation value among the evaluation values calculated for each evaluation model and the evaluation value for the first action indicated in the experience acquired in the first state indicated in the experience.
2 . The learning device according to claim 1 ,
wherein the one or more processors are configured to further execute the instructions to acquire a mini-batch of the experiences sampled based on the priority.
3 . A control system comprising:
a memory configured to store instructions; and
one or more processors configured to execute the instructions to:
associate a priority based on a magnitude of an error with an experience, the experience is a combination of a first state of a control subject, a first action of the control subject in the first state, a reward according to the first action in the first state, and a second state according to the first action in the first state, wherein the error is between
1) an evaluation value obtained based on a minimum evaluation value among evaluation values for the second action in the second state, which is the evaluation value in the first state indicated in the experience, and which is the evaluation value for the first action indicated in the experience, the evaluation value being obtained by applying the reward indicated in the experience, the second state indicated in the experience, and the second action calculated from the second state using a policy model to a plurality of evaluation models obtained by setting a plurality of parameter values in an evaluation model body, and
2) an evaluation value obtained for each evaluation model by applying the first state indicated in the experience and the first action indicated in the experience to a plurality of evaluation models obtained by setting a plurality of parameter values in the evaluation model body, and is a maximum evaluation value among the evaluation values for the first action in the first state;
acquire the experience based on the priority;
apply the second state indicated in the experience acquired and the second action calculated from the second state using the policy model to the plurality of evaluation models obtained by setting the plurality of parameter values in the evaluation model body, and calculating the evaluation value for the second action in the second state for each evaluation model;
update the evaluation model based on a minimum evaluation value among the evaluation values calculated for each evaluation model and the evaluation value for the first action indicated in the experience acquired in the first state indicated in the experience;
update the policy model using the evaluation model;
calculate a control value using the policy model; and
control a control subject on the basis of the control value.
4 . A learning method executed by a computer, the learning method comprising:
associating a priority based on a magnitude of an error with an experience, the experience is a combination of a first state of a control subject, a first action of the control subject in the first state, a reward according to the first action in the first state, and a second state according to the first action in the first state, wherein the error is between
1) an evaluation value obtained based on a minimum evaluation value among evaluation values for the second action in the second state, which is the evaluation value in the first state indicated in the experience, and which is the evaluation value for the first action indicated in the experience, the evaluation value being obtained by applying the reward indicated in the experience, the second state indicated in the experience, and the second action calculated from the second state using a policy model to a plurality of evaluation models obtained by setting a plurality of parameter values in an evaluation model body, and
2) an evaluation value obtained for each evaluation model by applying the first state indicated in the experience and the first action indicated in the experience to a plurality of evaluation models obtained by setting a plurality of parameter values in the evaluation model body, and is a maximum evaluation value among the evaluation values for the first action in the first state;
acquiring the experience based on the priority;
applying the second state indicated in the experience acquired and the second action calculated from the second state using the policy model to the plurality of evaluation models obtained by setting the plurality of parameter values in the evaluation model body, and calculating the evaluation value for the second action in the second state for each evaluation model; and
updating the evaluation model based on a minimum evaluation value among the evaluation values calculated for each evaluation model and the evaluation value for the first action indicated in the experience acquired in the first state indicated in the experience.