OFF-LINE LEARNING FOR AGENT CONTROL USING A REWARD PREDICTION MODEL
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for off-line learning using a reward prediction model. One of the methods includes obtaining robot experience data; training, on a first subset of the robot experience data, a reward prediction model that receives a reward input comprising an input observation and generates as output a reward prediction that is a prediction of a task-specific reward for the particular task that should be assigned to the input observation; processing experiences in the robot experience data using the trained reward prediction model to generate a respective reward prediction for each of the processed experiences; and training a policy neural network on (i) the processed experiences and (ii) the respective reward predictions for the processed experiences.
1 . A method comprising:
obtaining robot experience data characterizing robot interactions with an environment, the robot experience data comprising a plurality of experiences that each comprise (i) an observation characterizing a state of the environment and (ii) an action performed by a respective robot in response to the observation, wherein the experiences comprise:
expert experiences from episodes of a particular task being performed by an expert agent, and
unlabeled experiences;
training, on a first subset of the robot experience data, a reward prediction model that receives a reward input comprising an input observation and generates as output a reward prediction that is a prediction of a task-specific reward for the particular task that should be assigned to the input observation, wherein training the reward prediction model comprises optimizing an objective function that:
includes a first term that encourages the reward prediction model to assign, to observations from expert experiences, a first reward value that indicates that the particular task was completed successfully after the environment was in the state characterized by the observation, and
includes a second term that encourages the reward prediction model to assign, to observations from unlabeled experiences, a second reward value that indicates that the particular task was not completed successfully after the environment was in the state characterized by the observation;
processing experiences in the robot experience data using the trained reward prediction model to generate a respective reward prediction for each of the processed experiences; and
training a policy neural network on (i) the processed experiences and (ii) the respective reward predictions for the processed experiences, wherein the policy neural network is configured to receive a network input comprising an observation and to generate a policy output that defines a control policy for a robot performing the particular task.