GENERATING A MODEL OF A TARGET ENVIRONMENT BASED ON INTERACTIONS OF AN AGENT WITH SOURCE ENVIRONMENTS

1 . A method performed by one or more computers, the method comprising:

selecting actions to be performed by an agent to interact with a target environment over a sequence of time steps using an environment model,

wherein the environment model is parameterized by a set of environment model parameters defining a model of the target environment,

wherein for each time step, selecting an action to be performed by the agent at the time step comprises:

sampling current values of the set of environment model parameters in accordance with a probability distribution, wherein the probability distribution is derived from a set of interaction history data, wherein the set of interaction history data comprises:

(i) data characterizing interaction of the agent with the target environment at any preceding time steps in the sequence of time steps; and

(ii) data characterizing interaction of the agent with each of one or more source environments, wherein each of the source environments are different than the target environment;

generating, using the environment model and in accordance with the current values of the set of environment model parameters, a respective expected reward for each action in a set of actions that can be performed by the agent; and

selecting the action to be performed by the agent at the time step based on the expected rewards.

2 . The method of claim 1 , wherein sampling current values of the set of environment model parameters in accordance with the probability distribution comprises:

generating respective values of the set of environment model parameters based on the set of interaction history data at each sampling iteration in a sequence of sampling iterations; and

designating values of the set of environment model parameters generated at a particular sampling iteration in the sequence of sampling iterations as being sampled in accordance with the probability distribution.

3 . The method of claim 2 , wherein the values of the set of environment model parameters generated at each sampling iteration in the sequence of sampling iterations define a Markov chain with an invariant distribution given by the probability distribution.

4 . The method of claim 2 , further comprising obtaining relationship data that characterizes a respective relationship between the target environment and each of the source environments;

wherein at each sampling iteration, generating values of the set of environment model parameters based on the set of interaction history data comprises:

generating values of the set of environment model parameters based on: (i) the set of interaction history data, and (ii) the relationship data.

5 . The method of claim 4 , wherein the target environment and each of the source environments can be represented by respective graphs; and

wherein for each source environment, the relationship data defines a correspondence between one or more nodes in the graph representing the target environment with corresponding nodes in the graph representing the source environment.

6 . The method of claim 5 , wherein each node in the graph representing the target environment represents a respective element of the target environment.

7 . The method of claim 6 , wherein one or more nodes in the graph representing the target environment represent respective features of the target environment.

8 . The method of claim 6 , wherein a first node in the graph representing the target environment represents respective actions performed by the agent in the target environment.

9 . The method of claim 6 wherein a second node in the graph representing the target environment represents respective rewards received as a result of actions performed by the agent in the target environment.

10 . The method of claim 5 , wherein at each sampling iteration, generating values of the set of environment model parameters based on: (i) the set of interaction history data, and (ii) the relationship data comprises, for each of one or more environment model parameters:

designating one or more of the source environments as being relevant to the environment model parameter using the relationship data; and

generating the value of the environment model parameter based at least in part on data characterizing interaction of the agent with each of the one or more source environments designated as being relevant to the environment model parameter.

11 . The method of claim 10 , wherein designating one or more of the source environments as being relevant to the environment model parameter using the relationship data comprises:

determining that the environment model parameter is associated with a node in the graph representing the target environment;

determining, using the relationship data, that the node in the graph representing the target environment has a relationship with a corresponding node in a graph representing a source environment; and

in response, designating the source environment as being relevant to the environment model parameter.

12 . The method of claim 1 , further comprising, at each time step:

augmenting the set of interaction history data with data characterizing interaction of the agent with the target environment at the time step.

13 . The method of claim 1 , wherein for each time step, selecting the action to be performed by the agent at the time step based on the expected rewards comprises:

selecting an action having a highest expected reward from among the set of actions.

14 . The method of claim 1 , wherein for each time step, selecting the action to be performed by the agent at the time step based on the expected rewards comprises:

determining a probability distribution over the set of actions using the expected rewards; and

sampling an action in accordance with the probability distribution over the set of actions.

15 . The method of claim 1 , wherein for each source environment, the data characterizing the interaction of the agent with the source environment comprises a plurality of experience tuples, wherein each experience tuple comprises:

(i) an observation characterizing a state of the source environment,

(ii) an action performed by the agent in response to the observation; and

(iii) a reward received by the agent as a result of performing the action.

16 . A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

selecting actions to be performed by an agent to interact with a target environment over a sequence of time steps using an environment model,

wherein the environment model is parameterized by a set of environment model parameters defining a model of the target environment,

wherein for each time step, selecting an action to be performed by the agent at the time step comprises:

sampling current values of the set of environment model parameters in accordance with a probability distribution, wherein the probability distribution is derived from a set of interaction history data, wherein the set of interaction history data comprises:

(i) data characterizing interaction of the agent with the target environment at any preceding time steps in the sequence of time steps; and

(ii) data characterizing interaction of the agent with each of one or more source environments, wherein each of the source environments are different than the target environment;

generating, using the environment model and in accordance with the current values of the set of environment model parameters, a respective expected reward for each action in a set of actions that can be performed by the agent; and

selecting the action to be performed by the agent at the time step based on the expected rewards.

17 . The system of claim 16 , wherein sampling current values of the set of environment model parameters in accordance with the probability distribution comprises:

generating respective values of the set of environment model parameters based on the set of interaction history data at each sampling iteration in a sequence of sampling iterations; and

designating values of the set of environment model parameters generated at a particular sampling iteration in the sequence of sampling iterations as being sampled in accordance with the probability distribution.

18 . The system of claim 17 , wherein the values of the set of environment model parameters generated at each sampling iteration in the sequence of sampling iterations define a Markov chain with an invariant distribution given by the probability distribution.

19 . The system of claim 17 , wherein the operations further comprise obtaining relationship data that characterizes a respective relationship between the target environment and each of the source environments;

wherein at each sampling iteration, generating values of the set of environment model parameters based on the set of interaction history data comprises:

generating values of the set of environment model parameters based on: (i) the set of interaction history data, and (ii) the relationship data.

20 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

selecting actions to be performed by an agent to interact with a target environment over a sequence of time steps using an environment model,

wherein the environment model is parameterized by a set of environment model parameters defining a model of the target environment,

wherein for each time step, selecting an action to be performed by the agent at the time step comprises:

sampling current values of the set of environment model parameters in accordance with a probability distribution, wherein the probability distribution is derived from a set of interaction history data, wherein the set of interaction history data comprises:

(i) data characterizing interaction of the agent with the target environment at any preceding time steps in the sequence of time steps; and

(ii) data characterizing interaction of the agent with each of one or more source environments, wherein each of the source environments are different than the target environment;

generating, using the environment model and in accordance with the current values of the set of environment model parameters, a respective expected reward for each action in a set of actions that can be performed by the agent; and

selecting the action to be performed by the agent at the time step based on the expected rewards.