Training action selection neural networks using apprenticeship
An off-policy reinforcement learning actor-critic neural network system configured to select actions from a continuous action space to be performed by an agent interacting with an environment to perform a task. An observation defines environment state data and reward data. The system has an actor neural network which learns a policy function mapping the state data to action data. A critic neural network learns an action-value (Q) function. A replay buffer stores tuples of the state data, the action data, the reward data and new state data. The replay buffer also includes demonstration transition data comprising a set of the tuples from a demonstration of the task within the environment. The neural network system is configured to train the actor neural network and the critic neural network off-policy using stored tuples from the replay buffer comprising tuples both from operation of the system and from the demonstration transition data.
1. An off-policy reinforcement learning actor-critic neural network system, to select actions to be performed by an agent interacting with an environment to perform a task, the system comprising:
a first input to receive an observation comprising state data characterizing a state of the environment, and reward data representing a reward from operating with an action in the environment;
at least one actor neural network, coupled to receive the state data and configured to define a policy function mapping the state data to action data defining an action, wherein the at least one actor neural network has an output to provide the action data for the agent to perform the action, and wherein the environment transitions to a new state in response to the action;
at least one critic neural network, coupled to receive the action data, the state data, and return data derived from the reward data, and configured to define a value function which generates an error signal;
a replay buffer to store reinforcement learning transitions comprising operation transition data from operation of the system, wherein the operation transition data comprises tuples of said state data, said action data, said reward data and new state data representing said new state; and
a second input to receive training data defining demonstration transition data, the demonstration transition data comprising a set of said tuples from a demonstration of the task within the environment, wherein reinforcement learning transitions stored in the replay buffer further comprise the demonstration transition data; and
wherein the neural network system is configured to train the at least one actor neural network and the at least one critic neural network off-policy using the error signal and using stored tuples from the replay buffer comprising tuples from both the operation transition data and the demonstration transition data.
2. The system as claimed in claim 1 wherein said reward comprises a sparse reward which has a plurality of discrete values dependent upon the state of the environment.
3. The system as claimed in claim 1 wherein only a minority subset of states of the environment provide the reward.
4. The system as claimed in claim 1 , further comprising a sample selection system to sample the reinforcement learning transitions according to a sampling probability, wherein the sampling probability prioritizes sampling of the demonstration transition data tuples.
5. The system as claimed in claim 1 , wherein the return data comprises a combination of the reward data and values from the critic neural network obtained from an (n-1)-step forward rollout of actions selected using the actor neural network; and wherein the system is configured to employ at least two different values of n to train the network.
6. The system as claimed in claim 1 configured to train the critic neural network using return data which comprises a mix of 1-step and n-step returns.
7. The system as claimed in claim 1 comprising learning and target actor neural networks and learning and target critic neural networks, wherein the neural network system is configured to update the learning critic neural network off-policy using the error signal, wherein the error signal is derived from the target critic target neural network, the target actor neural network, and the stored tuples of both the operation transition data and the demonstration transition data; wherein the system is configured to update the learning actor neural network using a deterministic policy gradient comprising a product of a gradient of the output of the learning critic neural network and a gradient of the output of the learning actor neural network evaluated using the stored tuples of both the operation transition data and the demonstration transition data; and wherein the system is configured to, at intervals, update weights of the target actor neural network using the learning actor neural network and to update weights of the target critic neural network using the learning critic neural network.
8. The system as claimed in claim 1 wherein the training data comprises kinesthetic teaching data from manipulation of a mechanical system.
9. The system as claimed in claim 1 further comprising a safety controller to impose safety or other constraints on the action data.
10. The method of training a reinforcement learning system to select actions to be performed by an agent interacting with an environment to perform a task, the method comprising:
capturing training data from a demonstration of the task within the environment, the training data defining demonstration transition data for a series of demonstration transitions, each transition comprising state data characterizing a state of the environment, action data defining an action performed, reward data representing a reward from the action, and new state data representing a new state, wherein the action data defines one or more actions in a continuous action space, and wherein the environment transitions to the new state in response to the action;
storing the demonstration transition data in a replay buffer;
operating on the environment with an actor-critic system to generate operation transition data comprising operational examples of the state data, the action data, the reward data and the new state data;
storing the operation transition data in the replay buffer; and
sampling from the replay buffer to train the actor-critic system.
11. The method as claimed in claim 10 wherein the sampling prioritizes samples of the demonstration transition data over samples of the operation transition data.
12. The method as claimed in claim 10 wherein at least some of the demonstration transition data is retained in the replay buffer after storing the operation transition data.
13. The method as claimed in claim 10 wherein the reward is a sparse reward.
14. The method as claimed in claim 10 comprising training the actor-critic system using return data derived from a combination of the reward data and a discounted reward from a predicted succession of n-1 transitions forward from a current state of the environment, the method further comprising varying n or combining return data for different values of n.
15. The method as claimed in claim 10 comprising performing multiple training steps of the actor-critic system for each environment transition.
16. The method as claimed in claim 10 comprising operating on the environment during one or more operating time intervals and training the actor-critic system during one or more different, learning time intervals.
17. The method as claimed in claim 10 wherein the actor-critic system comprises learning and target actor neural networks and learning and target critic neural networks, and wherein training the actor-critic system comprises updating weights of the learning critic neural network using an error signal determined from the target actor neural network and target critic neural network, and updating weights of the learning actor neural network using an approximate policy gradient defining the performance of a policy function mapping the state data to the action data.
18. The method as claimed in claim 10 wherein the training data comprises kinesthetic teaching data from manipulation of a mechanical system.
19. The system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a reinforcement learning system to select actions to be performed by an agent interacting with an environment to perform a task, the operations comprising:
capturing training data from a demonstration of the task within the environment, the training data defining demonstration transition data for a series of demonstration transitions, each transition comprising state data characterizing a state of the environment, action data defining an action performed, reward data representing a reward from the action, and new state data representing a new state, wherein the action data defines one or more actions in a continuous action space, and wherein the environment transitions to the new state in response to the action;
storing the demonstration transition data in a replay buffer;
operating on the environment with an actor-critic system to generate operation transition data comprising operational examples of the state data, the action data, the reward data and the new state data;
storing the operation transition data in the replay buffer; and
sampling from the replay buffer to train the actor-critic system.
20. The system as claimed in claim 19 wherein the sampling prioritizes samples of the demonstration transition data over samples of the operation transition data.