Reinforcement learning using pseudo-counts
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining data identifying (i) a first observation characterizing a first state of the environment, (ii) an action performed by the agent in response to the first observation, and (iii) an actual reward received resulting from the agent performing the action in response to the first observation; determining a pseudo-count for the first observation; determining an exploration reward bonus that incentivizes the agent to explore the environment from the pseudo-count for the first observation; generating a combined reward from the actual reward and the exploration reward bonus; and adjusting current values of the parameters of the neural network using the combined reward.
1. A computer-implemented method for training a neural network used to select actions to be performed by an agent interacting with an environment, the computer-implemented method comprising:
obtaining data identifying a first observation characterizing a first state of the environment;
selecting, using the neural network, an action to be performed by the agent in response to the first observation;
controlling the agent to perform the selected action;
receiving an actual reward resulting from the agent performing the action in response to the first observation;
determining a pseudo-count for the first observation using a sequential density model which represents a likelihood that the first observation occurs given a sequence of previous observations, wherein the pseudo-count depends upon a number of previous occurrences of the first observation during the training of the neural network;
determining an exploration reward bonus that incentivizes the agent to explore the environment from the pseudo-count for the first observation, wherein the exploration reward bonus is lower when the pseudo-count is higher and vice-versa;
generating a combined reward from the actual reward and the exploration reward bonus; and
training the neural network by adjusting current values of parameters of the neural network using the combined reward.
2. The computer-implemented method of claim 1 , wherein adjusting the current values of the parameters comprises:
using the combined reward in place of the actual reward in performing an iteration of a reinforcement learning technique.
3. The computer-implemented method of claim 1 , wherein the neural network is configured to receive the first observation and generates an output that defines a probability distribution over possible actions, with a probability for each action being a probability that the action maximizes chances of the agent completing a task performed by the agent.
4. The computer-implemented method of claim 1 , wherein the neural network is configured to receive an observation-action pair which is the first observation and an action performed by the agent in response to the first observation, and to generate a Q-value for the observation-action pair that represents an estimated return resulting from the agent performing the action in response the observation in the observation-action pair.
5. The computer-implemented method of claim 1 , wherein generating the combined reward comprises summing the actual reward and the exploration reward bonus.
6. The computer-implemented method of claim 1 , wherein the exploration reward bonus RB satisfies:
RB
=
β
(
N
^
(
x
)
+
a
)
b
,
wherein x is the first observation, {circumflex over (N)}(x) is the pseudo-count for the first observation, a and b are constants, and β is a parameter selected by a parameter sweep.
7. The computer-implemented method of claim 1 , wherein the pseudo-count {circumflex over (N)}(x) for the first observation is of the following form:
N
^
n
(
x
)
=
ρ
n
(
x
)
(
1
-
ρ
n
′
(
x
)
)
ρ
n
′
(
x
)
-
ρ
n
(
x
)
,
wherein ρ n (x) is a value of a sequential density model for the first observation and ρ n ′(x) is a recoding probability for the first observation, wherein the recoding probability is a value of the sequential density model after observing a new occurrence of the first observation.
8. The computer-implemented method of claim 1 , wherein the sequential density model is a pixel-level density model.
9. The computer-implemented method of claim 1 , wherein the agent is a mechanical agent and the environment is a real-world environment, and wherein the neural network is trained as the agent explores the environment.
10. One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a neural network used to select actions to be performed by an agent interacting with an environment, the operations comprising:
obtaining data identifying a first observation characterizing a first state of the environment;
selecting, using the neural network, an action to be performed by the agent in response to the first observation;
controlling the agent to perform the selected action;
receiving an actual reward resulting from the agent performing the action in response to the first observation;
determining a pseudo-count for the first observation using a sequential density model which represents a likelihood that the first observation occurs given a sequence of previous observations, wherein the pseudo-count depends upon a number of previous occurrences of the first observation during the training of the neural network;
determining an exploration reward bonus that incentivizes the agent to explore the environment from the pseudo-count for the first observation, wherein the exploration reward bonus is lower when the pseudo-count is higher and vice-versa;
generating a combined reward from the actual reward and the exploration reward bonus; and
training the neural network by adjusting current values of parameters of the neural network using the combined reward.
11. The one or more non-transitory computer storage media of claim 10 , wherein the exploration reward bonus RB satisfies:
RB
=
β
(
N
^
(
x
)
+
a
)
b
,
wherein x is the first observation, {circumflex over (N)}(x) is the pseudo-count for the first observation, a and b are constants, and β is a parameter selected by a parameter sweep.
12. A system comprising one or more computers and one or more non-transitory computer storage media encoded with instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for training a neural network used to select actions to be performed by an agent interacting with an environment, the operations comprising:
obtaining data identifying a first observation characterizing a first state of the environment;
selecting, using the neural network, an action to be performed by the agent in response to the first observation;
controlling the agent to perform the selected action;
receiving an actual reward resulting from the agent performing the action in response to the first observation;
determining a pseudo-count for the first observation using a sequential density model which represents a likelihood that the first observation occurs given a sequence of previous observations, wherein the pseudo-count depends upon a number of previous occurrences of the first observation during the training of the neural network;
determining an exploration reward bonus that incentivizes the agent to explore the environment from the pseudo-count for the first observation, wherein the exploration reward bonus is lower when the pseudo-count is higher and vice-versa;
generating a combined reward from the actual reward and the exploration reward bonus; and
training the neural network by adjusting current values of parameters of the neural network using the combined reward.
13. The system of claim 12 , wherein adjusting the current values of the parameters comprises:
using the combined reward in place of the actual reward in performing an iteration of a reinforcement learning technique.
14. The system of claim 12 , wherein the neural network is configured to receive the first observation and generates an output that defines a probability distribution over possible actions, with a probability for each action being a probability that the action maximizes chances of the agent completing a task performed by the agent.
15. The system of claim 12 , wherein the neural network is configured to receive an observation-action pair which is the first observation and an action performed by the agent in response to the first observation, and to generate a Q-value for the observation-action pair that represents an estimated return resulting from the agent performing the action in response the observation in the observation-action pair.
16. The system of claim 12 , wherein generating the combined reward comprises summing the actual reward and the exploration reward bonus.
17. The system of claim 12 , wherein the exploration reward bonus RB satisfies:
RB
=
β
(
N
^
(
x
)
+
a
)
b
,
wherein x is the first observation, {circumflex over (N)}(x) is the pseudo-count for the first observation, a and b are constants, and β is a parameter selected by a parameter sweep.
18. The system of claim 12 , wherein the pseudo-count {circumflex over (N)}(x) for the first observation is of the following form:
N
^
n
(
x
)
=
ρ
n
(
x
)
(
1
-
ρ
n
′
(
x
)
)
ρ
n
′
(
x
)
-
ρ
n
(
x
)
,
wherein ρ n (x) is a value of a sequential density model for the first observation and ρ n ′(x) is a recoding probability for the first observation, wherein the recoding probability is a value of the sequential density model after observing a new occurrence of the first observation.
19. The system of claim 12 , wherein the sequential density model is a pixel-level density model.
20. The system of claim 12 , wherein the agent is a mechanical agent and the environment is a real-world environment, and wherein the neural network is trained as the agent explores the environment.