Exploration method and apparatus in multi-agent reinforcement learning
An exploration method used by an exploration apparatus in multi-agent reinforcement learning to collect training samples during the training process is provided. The exploration method includes calculating the influence of a selected action of each agent on the actions of other agents in a current state, calculating a linear sum of the value of a utility function representing the action value of each agent and the influence on the actions of the other agent calculated for the selected action of each agent, and obtaining a sample to be used for training an action policy of each agent by probabilistically selecting the action in which the linear sum is the maximum, and the random action.
1 . A method for exploring training samples used in learning action policy applied to actions of a plurality of agents included in a multi-agent reinforcement learning system, the plurality of agents configured to perform actions according to the learned action policy, and receive rewards for the actions performed, the method comprising:
calculating, by an exploration apparatus, in a current state and while a selected action of a target agent is fixed, a variance value of a joint value function for actions of other agents, the joint value function configured to estimate an expected return of cumulative reward by taking, as inputs, observation information, action information, and reward values of all agents;
calculating, by the exploration apparatus, a linear sum of a value of a utility function of the target agent and the variance value of the joint value function for the actions of the other agents calculated for the selected action of the target agent, the utility function configured, for each agent, to estimate an expected return of cumulative reward using only observation information of respective agents as an input; and
obtaining, by the exploration apparatus, a sample to be used for training an action policy of each agent by probabilistically selecting an action in which the linear sum is a maximum and a random action,
wherein the calculating of the variance value comprises:
determining, by the exploration apparatus, a variance of values of the utility function for respective actions of each of the agents, as a variance of an input; and
applying, by the exploration apparatus, the variance of the input to an approximate variance propagation algorithm configured to have a deep neural network function as a mapping function, to obtain a variance of an output of the deep neural network function, wherein variance propagation at each layer of the deep neural network function is performed using a Jacobian matrix for respective layers of the deep neural network function.
2 . The method of claim 1 , further comprising:
transitioning to a next state after each agent has taken the selected action in the current state;
reflecting a reward for the selected action to the utility function of a corresponding agent; and
performing the calculating of the variance value of the joint value function, the calculating of the linear sum, and the obtaining of the sample.
3 . An exploration apparatus for exploring training samples used in learning action policy applied to actions of a plurality of agents included in a multi-agent reinforcement learning system, the plurality of agents configured to perform actions according to the learned action policy, and receive rewards for the actions performed, the exploration apparatus comprising:
an information obtainer that obtains observation information and action information of each agent;
an agent influence calculator that calculates, in a current state and while a selected action of a target agent is fixed, a variance value of a joint value function for actions of other agents, the joint value function configured to estimate an expected return of cumulative reward by taking, as inputs, observation information, action information, and reward values of all agents; and
an action explorer that calculates a linear sum of the value of the utility function of the target agent and the variance value of the joint value function for the actions of the other agents calculated for a selected action of each agent, and obtains a sample to be used for training an action policy of each agent by probabilistically selecting an action in which the linear sum is a maximum, and a random action, the utility function configured, for each agent, to estimate an expected return of cumulative reward using only observation information of respective agents as an input,
wherein the action explorer is configured to:
determine a variance of values of the utility function for respective actions of each of the agents, as a variance of an input, and
apply the variance of the input to an approximate variance propagation algorithm configured to have a deep neural network function as a mapping function, to obtain a variance of an output of the deep neural network function, wherein variance propagation at each layer of the deep neural network function is performed using a Jacobian matrix for respective layers of the deep neural network function.
4 . The exploration apparatus of claim 3 , wherein the information obtainer, the agent influence calculator, and the action explorer are sequentially and repeatedly executed according to a state transition of each agent in a training process.