Dueling deep neural networks
View Patent ↗Systems, methods, and apparatus, including computer programs encoded on a computer storage medium, for selecting an actions from a set of actions to be performed by an agent interacting with an environment. In one aspect, the system includes a dueling deep neural network. The dueling deep neural network includes a value subnetwork, an advantage subnetwork, and a combining layer. The value subnetwork processes a representation of an observation to generate a value estimate. The advantage subnetwork processes the representation of the observation to generate an advantage estimate for each action in the set of actions. The combining layer combines the value estimate and the respective advantage estimate for each action to generate a respective Q value for the action. The system selects an action to be performed by the agent in response to the observation using the respective Q values for the actions in the set of actions.
1. A system for selecting actions from a set of actions to be performed by an agent interacting with an environment, the system comprising:
a dueling deep neural network implemented by one or more computers, the dueling deep neural network comprising:
one or more initial neural network layers configured to:
receive an observation characterizing a current state of the environment; and
process the observation using one or more initial neural network layers to generate a representation of the observation;
a value subnetwork configured to:
receive the representation of the observation characterizing the current state of the environment; and
process the representation of the observation to generate a value estimate, the value estimate being an estimate of an expected return resulting from the environment being in the current state;
an advantage subnetwork configured to:
receive the representation of the observation; and
process the representation of the observation to generate a respective advantage estimate for each action in the set of actions that is an estimate of a relative measure of the return resulting from the agent performing the action when the environment is in the current state relative to the return resulting from the agent performing other actions when the environment is in the current state; and
a combining layer configured to, for each action, combine the value estimate and the respective advantage estimate for the action to generate a respective Q value for the action, wherein the respective Q value is an estimate of an expected return resulting from the agent performing the action when the environment is in the current state, comprising:
determining a measure of central tendency of the respective advantage estimates for the actions in the set of actions;
determining a respective adjusted advantage estimate for the action by adjusting the respective advantage estimate for the action using the measure of central tendency; and
combining the respective adjusted advantage estimate for the action and the value estimate to determine the respective Q value for the action.
2. The system of claim 1 , wherein the system comprises one or more second computers and one or more storage devices storing instructions that when executed by the one or second more computers cause the one or more second computers to perform operations comprising:
selecting an action to be performed by the agent in response to the observation using the respective Q values for the actions in the set of actions.
3. The system of claim 1 , wherein the observation is an image and wherein the one or more initial neural network layers are convolutional neural network layers.
4. The system of claim 1 , wherein the value subnetwork has a first set of parameters and the advantage subnetwork has a second, different set of parameters.
5. The system of claim 2 , wherein selecting an action to be performed by the agent in response to the observation using the respective Q values for the actions in the set of actions comprises:
selecting an action having a highest Q value as the action to be performed by the agent.
6. The system of claim 2 , wherein selecting an action to be performed by the agent in response to the observation using the respective Q values for the actions in the set of actions comprises:
selecting a random action from the set of actions with probability ε; and
selecting an action having a highest Q value with probability 1−ε.
7. A method for selecting actions from a set of actions to be performed by an agent interacting with an environment using a dueling deep neural network comprising a value subnetwork and an advantage subnetwork, the method comprising:
receiving an observation characterizing a current state of the environment; and
processing the observation using one or more initial neural network layers to generate a representation of the observation;
obtaining the representation of the observation characterizing the current state of the environment;
processing the representation of the observation using the value subnetwork, wherein the value subnetwork is configured to:
receive the representation of the observation; and
process the representation of the observation to generate a value estimate, the value estimate being an estimate of an expected return resulting from the environment being in the current state;
processing the representation of the observation using the advantage subnetwork, wherein the advantage subnetwork is configured to:
receive the representation of the observation; and
process the representation of the observation to generate a respective advantage estimate for each action in the set of actions that is an estimate of a relative measure of the return resulting from the agent performing the action when the environment is in the current state relative to the return resulting from the agent performing other actions when the environment is in the current state;
for each action, combining the value estimate and the respective advantage estimate for the action to generate a respective Q value for the action, wherein the respective Q value is an estimate of an expected return resulting from the agent performing the action when the environment is in the current state, comprising:
determining a measure of central tendency of the respective advantage estimates for the actions in the set of actions;
determining a respective adjusted advantage estimate for the action by adjusting the respective advantage estimate for the action using the measure of central tendency; and
combining the respective adjusted advantage estimate for the action and the value estimate to determine the respective Q value for the action; and
selecting an action to be performed by the agent in response to the observation using the respective Q values for the actions in the set of actions.
8. The method of claim 7 , wherein the observation is an image and wherein the one or more initial neural network layers are convolutional neural network layers.
9. The method of claim 7 , wherein the value subnetwork has a first set of parameters and the advantage subnetwork has a second, different set of parameters.
10. The method of claim 7 , wherein selecting an action to be performed by the agent in response to the observation using the respective Q values for the actions in the set of actions comprises:
selecting an action having a highest Q value as the action to be performed by the agent.
11. The method of claim 7 , wherein selecting an action to be performed by the agent in response to the observation using the respective Q values for the actions in the set of actions comprises:
selecting a random action from the set of actions with probability ε; and
selecting an action having a highest Q value with probability 1−ε.
12. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for selecting actions from a set of actions to be performed by an agent interacting with an environment using a dueling deep neural network comprising a value subnetwork and an advantage subnetwork, the operations comprising:
receiving an observation characterizing a current state of the environment; and
processing the observation using one or more initial neural network layers to generate a representation of the observation;
obtaining the representation of the observation characterizing the current state of the environment;
processing the representation of the observation using the value subnetwork, wherein the value subnetwork is configured to:
receive the representation of the observation; and
process the representation of the observation to generate a value estimate, the value estimate being an estimate of an expected return resulting from the environment being in the current state;
processing the representation of the observation using the advantage subnetwork, wherein the advantage subnetwork is configured to:
receive the representation of the observation; and
process the representation of the observation to generate a respective advantage estimate for each action in the set of actions that is an estimate of a relative measure of the return resulting from the agent performing the action when the environment is in the current state relative to the return resulting from the agent performing other actions when the environment is in the current state;
for each action, combining the value estimate and the respective advantage estimate for the action to generate a respective Q value for the action, wherein the respective Q value is an estimate of an expected return resulting from the agent performing the action when the environment is in the current state comprising:
determining a measure of central tendency of the respective advantage estimates for the actions in the set of actions;
determining a respective adjusted advantage estimate for the action by adjusting the respective advantage estimate for the action using the measure of central tendency; and
combining the respective adjusted advantage estimate for the action and the value estimate to determine the respective Q value for the action; and
selecting an action to be performed by the agent in response to the observation using the respective Q values for the actions in the set of actions.