Actor ensemble for continuous control
A method of training a reinforcement learning agent to output actions from a continuous action space, comprising: providing an actor ensemble that includes a plurality of actor neural networks that each output a respective action from the continuous action space in response to an observed state of an environment; providing a critic neural network that approximates a state-action value function indicating an impact of an action on the environment based on a reward from the environment and the observed state of the environment; training the actor ensemble and the critic neural network to maximize a state-action value from the state-action value function over successive time steps by, in each time step: selecting from the respective actions output by the plurality of actor neural networks the action that will provide a best state-action value from the state-action value function; applying the selected action to the environment; based on an observed state of the environment of in response to the selected action, determine a gradient ascent for the plurality of actor neural networks for updating the parameters of the plurality of actor neural networks and determine a gradient descent for the critic neural network for updating the parameters of the critic neural network.
1 . A method of using a reinforcement learning (RL) agent of an artificial intelligence (AI) system, the RL agent comprising an action selector and a critic, the method comprising the steps of:
(a-1) initializing parameters of each actor neural network among a plurality of actor neural networks of the RL agent, wherein the plurality of actor neural networks forms an actor ensemble, and wherein the parameters of each actor neural network of the plurality of actor neural networks in the actor ensemble are initialized differently;
(a-2) at each actor neural network among the plurality of actor neural networks in the actor ensemble of the RL agent, receiving a current state, (s), of an environment for a current time step and outputting a continuous action, (a), for the current state based on a deterministic policy approximated by the respective actor neural network, thereby outputting a plurality of continuous actions;
(b) at a critic neural network of the critic of the RL agent, receiving the current state of the environment and the continuous action output by each respective actor neural network in the actor ensemble and outputting a state-action value for the current state and the respective continuous action based on a state-action value function approximated by the critic neural network, thereby outputting a plurality of state-action values, each state-action value, among the plurality of state-action values, associated with a respective continuous action among the plurality of continuous actions;
(c) selecting, at the action selector of the RL agent, a particular continuous action from among the plurality of continuous actions, wherein the selected continuous action is associated with a state-action value that is maximum among the plurality of state-action values;
(d) causing the AI system comprising the RL agent to carry out the selected continuous action in the environment;
(e) generating an experience tuple (s, a, r, s′) comprising the current state of the environment(s), the selected continuous action (a), a reward provided by the environment (r), and a subsequent state of the environment (s′);
(f) storing the experience tuple (s, a, r, s′) in a replay buffer comprising a set of experience tuples;
(g) sampling the replay buffer to obtain a batch of tuples from the set of experience tuples;
(h) determining, by an actor parameter update determiner connected to receive the output of each actor neural network and the output of the critic neural network, a respective update for the parameters of each respective actor neural network of the plurality of actor neural networks in the actor ensemble based on the batch of tuples;
(i) providing, by the actor parameter update determiner, to each actor neural network among the plurality of actor neural networks in the actor ensemble, the respective update;
(j) training the plurality of actor neural networks in the actor ensemble by causing the respective update for the parameters of each respective actor neural network to be applied;
(k) determining, by a critic parameter update determiner connected to receive the output of the critic neural network, an update for the parameters of the critic neural network based on the batch of tuples, a determined state-action value, and a target state-action value (y);
(l) providing to the critic neural network, by the critic parameter update determiner, the update for the parameters of the critic neural network; and
(m) training the critic neural network by causing the update for the parameters of the critic neural network to be applied.
2 . The method of claim 1 , further comprising repeating steps (a-2) to (m) for a predetermined number of time steps.
3 . The method of claim 1 , further comprising:
at the critic neural network, performing a look-ahead tree search and backup process to predict the state-action value for the state and the respective continuous action.
4 . The method of claim 1 , wherein the critic neural network represents a value prediction model.
5 . The method of claim 1 , wherein the critic neural network represents a transition model.
6 . The method of claim 1 , wherein the update for the parameters of the critic neural network is determined based on a loss value for the critic neural network based on a temporal-difference (TD) error of the critic neural network.
7 . The method of claim 6 , wherein the TD error of the critic neural network is represented as
1
2
(
r
t
+
1
+
y
max
i
∈
{
1
,
…
,
N
}
Q
(
s
t
+
1
,
μ
i
(
s
t
+
1
)
)
-
Q
(
s
t
,
a
t
)
)
2
wherein s t is the current time step, a t is a selected continuous action at the current time step, s t+1 is a subsequent time step, Q(s t ,a t ) is a state-action value of a state-action value function for a state of the environment and the selected continuous action at the current time step, μ i (s t+1 ) is a selected continuous action at the subsequent time step, r t+1 is a reward for the selected continuous action at the subsequent time step, y is the target state-action value, ymax i∈{1, . . . , N} is a maximum of the target state-action value for the plurality of continuous actions, and N is a number of the plurality of continuous actions.
8 . The method of claim 7 , wherein the critic parameter update determiner trains the critic neural network to minimize the TD error, wherein the TD error is determined as a difference between the determined state-action value
(
f
q
d
(
z
,
a
)
)
and the target state-action value (y), and to ground a predicted reward ({circumflex over (r)}) for a current latent state (z) for the current time step and a current action (a) for the current time step by subtracting a received reward (r) from the predicted reward ({circumflex over (r)}), wherein the current latent state (z) is an n-dimensional latent state transformed from the current state(s) of the current time step via an encoding function (ƒ enc ).
9 . The method of claim 8 , wherein the critic parameter update determiner determines a per time step update (Δθ Q ) for the critic neural network parameters, θ Q , as represented by the following equation
θ
Q
←
θ
Q
-
α
∇
θ
Q
(
1
2
(
f
q
d
(
z
,
a
)
-
y
)
2
+
1
2
(
r
ˆ
-
r
)
2
)
,
wherein z′ is a subsequent latent state for a state of the subsequent time step determined by the following equation
y
′
←
{
0
if
s
is
terminal
max
i
f
q
d
(
z
′
,
f
μ
i
(
z
′
)
)
otherwise
,
y
←
r
+
γ
y
′
,
y is a target state-action value for the subsequent latent state (z′) and a best action is selected from the plurality of actor neural networks in the actor ensemble using the critic neural network by performing a look-ahead tree search, a is an action from the transition in the batch of tuples, r is an immediate reward received by the critic parameter update determiner from the replay buffer, and y′ is a maximum state-action value of the subsequent latent state (z′).
10 . The method of claim 9 , wherein the update for the parameters of the critic neural network is determined using gradient descent, the critic neural network of the critic being trained by taking one gradient descent algorithm step at each time, t, to minimize the TD error, wherein the critic parameter update determiner determines a gradient for the critic neural network parameters, θ Q , through use of the following equation
∇
θ
Q
(
1
2
(
f
q
d
(
z
,
a
)
-
y
)
2
+
1
2
(
r
ˆ
-
r
)
2
)
.
11 . The method of claim 10 , wherein the respective update for the parameters of each respective actor neural network of the plurality of actor neural networks in the actor ensemble is determined using a gradient ascent algorithm, the plurality of actor neural networks being trained, in parallel, to optimize the actor ensemble parameters, θ μ , via the gradient ascent algorithm, to maximize each state-action value associated with the action output from each actor neural network to find a global maximum.
12 . The method of claim 11 , wherein the actor parameter update determiner may determine a per time step update (Δθ μ ) to the actor ensemble parameters, θ μ , as represented by the following equation:
θ
μ
←
θ
μ
+
β
∇
θ
μ
∑
i
=
1
N
∇
b
f
q
d
(
z
,
b
)
|
b
=
f
μ
i
(
z
)
∇
θ
μ
f
μ
i
(
z
)
wherein
∇
b
f
q
d
(
z
,
b
)
|
b
=
f
μ
i
(
z
)
∇
θ
μ
f
μ
i
(
z
)
is used by the actor parameter update determiner to determine a gradient for the actor ensemble parameters θ μ ,
∇
b
f
q
d
(
z
,
b
)
|
b
=
f
μ
i
(
z
)
is a respective gradient for a state-action value function
f
q
d
(
z
,
b
)
|
b
=
f
μ
i
(
z
)
for a corresponding actor neural network at the current latent state (z), and ∇ θ μ θ μ i (z) is a respective gradient function for the action function (ƒ μ i (z)) for corresponding actor neural network at the current latent state (z).
13 . The method of claim 1 , wherein the AI system is coupled to a control system of a vehicle, robot or machine interacting with the environment to accomplish a particular task, wherein each continuous action in the plurality of continuous actions corresponds to a set of control inputs to control the vehicle, robot or machine to accomplish a particular task, and wherein step (c) and (d) comprise:
(c) selecting, at the action selector of the RL agent, a particular set of control inputs from among a plurality of set of control inputs corresponding to the plurality of continuous action, wherein the selected set of control inputs is associated with a state-action value that is maximum among the plurality of state-action values; and
(d) causing the control system of the vehicle, robot or machine to execute the control inputs to control the vehicle, robot or machine interacting with the environment to accomplish the particular task.
14 . The method of claim 1 , wherein the environment is a simulation environment generated by a computing system and the AI system is coupled to the computing system, wherein each continuous action in the plurality of continuous actions corresponds to a set of control inputs to control a simulated user, vehicle, robot or machine in the simulation environment to accomplish a particular task, and wherein step (c) and (d) comprise:
(c) selecting, at the action selector of the RL agent, a particular set of control inputs from among a plurality of set of control inputs corresponding to the plurality of continuous action, wherein the selected set of control inputs is associated with a state-action value that is maximum among the plurality of state-action values; and
(d) causing the computing system to execute the control inputs to control the simulated user, vehicle, robot or machine in the simulation environment to accomplish the particular task.
15 . A processing unit comprising:
a non-transitory computer-readable medium having tangibly stored thereon executable instructions;
a processor configured, by the executable instructions, to use train a reinforcement learning (RL) agent of an artificial intelligence (AI) system, the RL agent comprising an action selector and a critic, wherein the executable instructions, in response to execution by the processor, cause the processor to train the (RL) agent by:
(a-1) initializing parameters of each actor neural network among a plurality of actor neural networks of the RL agent, wherein the plurality of actor neural networks forms an actor ensemble, and wherein the parameters of each actor neural network of the plurality of actor neural networks in the actor ensemble are initialized differently;
(a-2) at each actor neural network among the plurality of actor neural networks in the actor ensemble of the RL agent, receiving a current state, (s), of an environment for a current time step and outputting a continuous action, (a), for the current state based on a deterministic policy approximated by the respective actor neural network, thereby outputting a plurality of continuous actions;
(b) at a critic neural network of the critic of the RL agent, receiving the current state of the environment and the continuous action output by each respective actor neural network in the actor ensemble and outputting a state-action value for the current state and the respective continuous action based on a state-action value function approximated by the critic neural network, thereby outputting a plurality of state-action values, each state-action value, among the plurality of state-action values, associated with a respective continuous action among the plurality of continuous actions;
(c) selecting, at the action selector of the RL agent, a particular continuous action from among the plurality of continuous actions, wherein the selected continuous action is associated with a state-action value that is maximum among the plurality of state-action values;
(d) causing the AI system comprising the RL agent to carry out the selected continuous action in the environment;
(e) generating an experience tuple (s, a, r, s′) comprising the current state of the environment(s), the selected continuous action (a), a reward provided by the environment (r), and a subsequent state of the environment (s′);
(f) storing the experience tuple (s, a, r, s′) in a replay buffer comprising a set of experience tuples;
(g) sampling the replay buffer to obtain a batch of tuples from the set of experience tuples;
(h) determining, by an actor parameter update determiner connected to receive the output of each actor neural network and the output of the critic neural network, a respective update for the parameters of each respective actor neural network of the plurality of actor neural networks in the actor ensemble based on the batch of tuples;
(i) providing, by the actor parameter update determiner, to each actor neural network among the plurality of actor neural networks in the actor ensemble, the respective update;
(j) training the plurality of actor neural networks in the actor ensemble by causing the respective update for the parameters of each respective actor neural network to be applied;
(k) determining, by a critic parameter update determiner connected to receive the output of the critic neural network, an update for the parameters of the critic neural network based on the batch of tuples, a determined state-action value, and a target state-action value (y);
(l) providing to the critic neural network, by the critic parameter update determiner, the update for the parameters of the critic neural network; and
(m) training the critic neural network by causing the update for the parameters of the critic neural network to be applied.
16 . The processing unit of claim 15 , wherein the processor is further configured, by execution of the executable instructions, to train the RL agent by repeating steps (a-2) to (m) for a predetermined number of time steps.
17 . The processing unit of claim 15 , wherein the processor is further configured, by execution of the executable instructions, to train the RL agent by, at the critic neural network, performing a look-ahead tree search and backup process to predict the state-action value for the state and the respective continuous action.
18 . The processing unit of claim 15 , wherein the critic neural network represents a value prediction model.
19 . The processing unit of claim 15 , wherein the critic neural network represents a transition model.
20 . A non-transitory computer-readable medium comprising executable instructions that, in response to execution by a processor, cause the processor to train use a reinforcement learning (RL) agent of an artificial intelligence (AI) system, the RL agent comprising an action selector and a critic, by:
(a-1) initializing parameters of each actor neural network among a plurality of actor neural networks of the RL agent, wherein the plurality of actor neural networks forms an actor ensemble, and wherein the parameters of each actor neural network of the plurality of actor neural networks in the actor ensemble are initialized differently;
(a-2) at each actor neural network among the plurality of actor neural networks in the actor ensemble of the RL agent, receiving a current state, (s), of an environment for a current time step and outputting a continuous action, (a), for the current state based on a deterministic policy approximated by the respective actor neural network, thereby outputting a plurality of continuous actions;
(b) at a critic neural network of the critic of the RL agent, receiving the current state of the environment and the continuous action output by each respective actor neural network in the actor ensemble and outputting a state-action value for the current state and the respective continuous action based on a state-action value function approximated by the critic neural network, thereby outputting a plurality of state-action values, each state-action value, among the plurality of state-action values, associated with a respective continuous action among the plurality of continuous actions;
(c) selecting, at the action selector of the RL agent, a particular continuous action from among the plurality of continuous actions, wherein the selected continuous action is associated with a state-action value that is maximum among the plurality of state-action values;
(d) causing the AI system comprising the RL agent to carry out the selected continuous action in the environment;
(e) generating an experience tuple (s, a, r, s′) comprising the current state of the environment(s), the selected continuous action (a), a reward provided by the environment (r), and a subsequent state of the environment (s′);
(f) storing the experience tuple (s, a, r, s′) in a replay buffer comprising a set of experience tuples;
(g) sampling the replay buffer to obtain a batch of tuples from the set of experience tuples;
(h) determining, by an actor parameter update determiner connected to receive the output of each actor neural network and the output of the critic neural network, a respective update for the parameters of each respective actor neural network of the plurality of actor neural networks in the actor ensemble based on the batch of tuples;
(i) providing, by the actor parameter update determiner, to each actor neural network among the plurality of actor neural networks in the actor ensemble, the respective update;
(j) training the plurality of actor neural networks in the actor ensemble by causing the respective update for the parameters of each respective actor neural network to be applied;
(k) determining, by a critic parameter update determiner connected to receive the output of the critic neural network, an update for the parameters of the critic neural network based on the batch of tuples, a determined state-action value, and a target state-action value (y);
(l) providing to the critic neural network, by the critic parameter update determiner, the update for the parameters of the critic neural network; and
(m) training the critic neural network by causing the update for the parameters of the critic neural network to be applied.
21 . The non-transitory computer-readable medium of claim 20 , wherein the instructions, to train the RL agent by, at the critic neural network, performing a look-ahead tree search and backup process to predict the state-action value for the state and the respective continuous action.