Device and method to improve learning of a policy for robots
A computer-implemented method for for learning a policy. The method includes: recording at least an episode of interactions of the agent with its environment following policy and adding the recorded episode to a set of training data; optimizing a transition dynamics model based on the training data such that the transition dynamics model predicts the next states of the environment depending on the states and actions contained in the training data; optimizing policy parameters based on the training data and the transition dynamics model by optimizing a reward. In the method, the transition dynamics model comprises a first model characterizing the global model and a second model characterizing a correction model, which is configured to correct outputs of the first model.
1. A computer-implemented method for operating an agent depending on a learned policy obtained by:
initializing a policy and a transition dynamics model which predicts a next state of an environment and/or of the agent; and
repeating the following steps until a termination condition is fulfilled:
recording at least an episode of interactions of the agent with the environment following the policy and adding the recorded episode to a set of training data,
optimizing the transition dynamics model based on the training data such that the transition dynamics model predicts next states of the environment depending on states and actions contained in the training data, and
optimizing policy parameters of the policy based on the training data and the transition dynamics model by optimizing a reward over at least one episode by following the policy;
wherein the transition dynamics model includes a first model representing a learned model of the environment and a correction model which is configured to correct errors of the first model;
wherein the method includes:
sensing the environment using a sensor of the agent;
determining a current state depending on the sensed environment;
determining, using the learning policy, an action for the agent depending on the current state;
carrying out the determined action, by the agent, wherein the correction model is optimized such that, when actions are selected as done when recording the episodes of the training data, then a sequence of states predicted by the transition dynamics model will be equal to the recorded states of the training data.
2. The method according to claim 1 , wherein the correction model is selected by minimizing a difference between an output of the correction model and a difference between the recorded state of the training data and the predicted state by the first model.
3. A computer-implemented method for operating an agent depending on a learned policy obtained by:
initializing a policy and a transition dynamics model which predicts a next state of an environment and/or of the agent; and
repeating the following steps until a termination condition is fulfilled:
recording at least an episode of interactions of the agent with the environment following the policy and adding the recorded episode to a set of training data,
optimizing the transition dynamics model based on the training data such that the transition dynamics model predicts next states of the environment depending on states and actions contained in the training data, and
optimizing policy parameters of the policy based on the training data and the transition dynamics model by optimizing a reward over at least one episode by following the policy;
wherein the transition dynamics model includes a first model representing a learned model of the environment and a correction model which is configured to correct errors of the first model;
wherein the method includes:
sensing the environment using a sensor of the agent;
determining a current state depending on the sensed environment;
determining, using the learning policy, an action for the agent depending on the current state;
carrying out the determined action, by the agent, wherein the correction model is dependent on a state or a time, wherein the time characterizes a time span elapsed since a beginning of a respective episode.
4. The method according to claim 3 , wherein the environment is deterministic and the correction model is dependent on the time.
5. A computer-implemented method for operating an agent depending on a learned policy obtained by:
initializing a policy and a transition dynamics model which predicts a next state of an environment and/or of the agent; and
repeating the following steps until a termination condition is fulfilled:
recording at least an episode of interactions of the agent with the environment following the policy and adding the recorded episode to a set of training data,
optimizing the transition dynamics model based on the training data such that the transition dynamics model predicts next states of the environment depending on states and actions contained in the training data, and
optimizing policy parameters of the policy based on the training data and the transition dynamics model by optimizing a reward over at least one episode by following the policy;
wherein the transition dynamics model includes a first model representing a learned model of the environment and a correction model which is configured to correct errors of the first model;
wherein the method includes:
sensing the environment using a sensor of the agent;
determining a current state depending on the sensed environment;
determining, using the learning policy, an action for the agent depending on the current state;
carrying out the determined action, by the agent, wherein the correction model is a probabilistic function, and wherein the probabilistic function is optimized by approximate inference.
6. A computer-implemented method for operating an agent depending on a learned policy obtained by:
initializing a policy and a transition dynamics model which predicts a next state of an environment and/or of the agent; and
repeating the following steps until a termination condition is fulfilled:
recording at least an episode of interactions of the agent with the environment following the policy and adding the recorded episode to a set of training data,
optimizing the transition dynamics model based on the training data such that the transition dynamics model predicts next states of the environment depending on states and actions contained in the training data, and
optimizing policy parameters of the policy based on the training data and the transition dynamics model by optimizing a reward over at least one episode by following the policy;
wherein the transition dynamics model includes a first model representing a learned model of the environment and a correction model which is configured to correct errors of the first model;
wherein the method includes:
sensing the environment using a sensor of the agent;
determining a current state depending on the sensed environment;
determining, using the learning policy, an action for the agent depending on the current state;
carrying out the determined action, by the agent, wherein the correction model is optimized jointly with the first model.
7. The method according to claim 6 , wherein the agent is an at least partially autonomous robot and/or a manufacturing machine and/or an access control system.
8. The method according to claim 6 , carrying out the determined action, by the agent, wherein for optimizing the transition dynamics model, after optimizing the first model on the training data, the correction model is selected such that error of the first model is minimized for actions selected from the policy on the training data.
9. A machine-readable storage medium on which is stored a computer program for learning a policy for an agent, the computer program, when executed by a computer, causing the computer to perform the following steps:
initializing a policy and a transition dynamics model which predicts a next state of an environment and/or of the agent; and
repeating the following steps until a termination condition is fulfilled:
recording at least an episode of interactions of the agent with the environment following the policy and adding the recorded episode to a set of training data,
optimizing the transition dynamics model based on the training data such that the transition dynamics model predicts next states of the environment depending on states and actions contained in the training data, and
optimizing policy parameters of the policy based on the training data and the transition dynamics model by optimizing a reward over at least one episode by following the policy;
wherein the transition dynamics model includes a first model representing a learned model of the environment and a correction model which is configured to correct errors of the first model, wherein the computer program, when executed by the computer, causing the computer to perform the following further steps:
sensing the environment using a sensor of the agent;
determining a current state depending on the sensed environment;
determining, using the learning policy, an action for the agent depending on the current state;
carrying out the determined action, by the agent wherein the correction model is optimized jointly with the first model.
10. A control system for operating an actuator, control system comprising:
a policy trained by:
initializing a policy and a transition dynamics model which predicts a next state of an environment and/or of the agent; and
repeating the following steps until a termination condition is fulfilled:
recording at least an episode of interactions of the agent with the environment following the policy and adding the recorded episode to a set of training data,
optimizing the transition dynamics model based on the training data such that the transition dynamics model predicts next states of the environment depending on states and actions contained in the training data, and
optimizing policy parameters of the policy based on the training data and the transition dynamics model by optimizing a reward over at least one episode by following the policy;
wherein the transition dynamics model includes a first model representing a learned model of the environment and a correction model which is configured to correct errors of the first model;
wherein the control system is configured to operate the actuator in accordance with an output of the policy, wherein the correction model is optimized jointly with the first model.