Granted Patent B2

US 12,265,795 · App. 18/649,774 · Granted Apr 1, 2025

Inventors: Karl Moritz Hermann (Berlin, DE); Philip Blunsom (Oxford, GB); Felix George Hill (London, GB)

Assignee: DeepMind Technologies Limited

G06F40/30G06F17/16G06N3/08

Abstract

Claims (69)

1. A method performed by one or more computers for selecting actions to be performed by an agent interacting with an environment, the method comprising:

at each of a plurality of time steps:

receiving a current text string in a natural language that expresses information about a current task being performed by the agent;

receiving a current observation characterizing a current state of the environment;

processing an input comprising the current text string and the current observation using a neural network to generate an action selection output, the processing comprising:

combining, by the neural network and in accordance with values of a set of neural network parameters, the current text string and the current observation to produce a combined embedding; and

generating, by the neural network and in accordance with the values of the set of neural network parameters, the action selection output based on the combined embedding; and

selecting an action to be performed by the agent at the time step based on the action selection output;

wherein the neural network has been trained from end-to-end using a machine learning training technique.

2. The method of claim 1 , further comprising:

receiving, at each of the plurality of time steps, a current reward as a result of the agent performing the action in response to the current observation; and

training the neural network from end-to-end using reinforcement learning based on the rewards received over the plurality of time steps.

3. The method of claim 1 , wherein combining, by the neural network and in accordance with the values of the set of neural network parameters, the current text string and the current observation to produce the combined embedding comprises:

processing the current text string using a language encoder model of the neural network to generate a current text embedding of the current text string;

processing the current observation using an observation encoder neural network of the neural network to generate a current observation embedding of the current observation; and

combining the current observation embedding and the current text embedding to generate the combined embedding.

4. The method of claim 1 , wherein generating, by the neural network and in accordance with the values of the set of neural network parameters, the action selection output based on the combined embedding comprises:

processing the combined embedding using an action selection neural network of the neural network to generate the action selection output.

5. The method of claim 3 , wherein the language encoder model is a recurrent neural network.

6. The method of claim 3 , wherein the language encoder model is a bag-of-words encoder.

7. The method of claim 3 , wherein the current observation embedding is a feature matrix of the current observation, and wherein the current text embedding is a feature vector of the current text string.

8. The method of claim 7 , wherein combining the current observation embedding and the current text embedding comprises:

flattening the feature matrix of the current observation; and

concatenating the flattened feature matrix and the feature vector of the current text string.

9. The method of claim 1 , wherein at each of the plurality of time steps, the current text string is a natural language instruction for the agent for performing the current task.

10. The method of claim 1 , wherein at each of the plurality of time steps:

the action selection output defines a probability distribution over possible actions to be performed by the agent; and

selecting the action to be performed by the agent comprises:

sampling an action from the probability distribution or selecting an action having a highest probability according to the probability distribution.

11. The method of claim 1 , wherein at each of the plurality of time steps:

the action selection output comprises, for each of a plurality of possible actions to be performed by the agent, a respective Q value that is an estimate of a return resulting from the agent performing the possible action in response to the current observation; and

selecting the action to be performed by the agent comprises:

selecting an action having a highest Q value.

12. The method of claim 1 , wherein at each of the plurality of time steps:

the action selection output identifies a best possible action to be performed by the agent in response to the current observation; and

selecting the action to be performed by the agent comprises:

selecting the best possible action.

13. The method of claim 1 , wherein the current text string is the same for each observation received during the performance of the current task.

14. The method of claim 1 , wherein the current text string is different from a preceding text string received during the performance of the current task.

15. A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for selecting actions to be performed by an agent interacting with an environment, the operations comprising:

at each of a plurality of time steps:

receiving a current text string in a natural language that expresses information about a current task being performed by the agent;

receiving a current observation characterizing a current state of the environment;

processing an input comprising the current text string and the current observation using a neural network to generate an action selection output, the processing comprising:

combining, by the neural network and in accordance with values of a set of neural network parameters, the current text string and the current observation to produce a combined embedding; and

generating, by the neural network and in accordance with the values of the set of neural network parameters, the action selection output based on the combined embedding; and

selecting an action to be performed by the agent at the time step based on the action selection output;

wherein the neural network has been trained from end-to-end using a machine learning training technique.

16. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for selecting actions to be performed by an agent interacting with an environment, the operations comprising:

at each of a plurality of time steps:

receiving a current text string in a natural language that expresses information about a current task being performed by the agent;

receiving a current observation characterizing a current state of the environment;

processing an input comprising the current text string and the current observation using a neural network to generate an action selection output, the processing comprising:

combining, by the neural network and in accordance with values of a set of neural network parameters, the current text string and the current observation to produce a combined embedding; and

generating, by the neural network and in accordance with the values of the set of neural network parameters, the action selection output based on the combined embedding; and

selecting an action to be performed by the agent at the time step based on the action selection output;

wherein the neural network has been trained from end-to-end using a machine learning training technique.

17. The non-transitory computer storage media of claim 16 , wherein the operations further comprise:

receiving, at each of the plurality of time steps, a current reward as a result of the agent performing the action in response to the current observation; and

training the neural network from end-to-end using reinforcement learning based on the rewards received over the plurality of time steps.

18. The non-transitory computer storage media of claim 16 , wherein combining, by the neural network and in accordance with the values of the set of neural network parameters, the current text string and the current observation to produce the combined embedding comprises:

processing the current text string using a language encoder model of the neural network to generate a current text embedding of the current text string;

processing the current observation using an observation encoder neural network of the neural network to generate a current observation embedding of the current observation; and

combining the current observation embedding and the current text embedding to generate the combined embedding.

19. The non-transitory computer storage media of claim 17 , wherein generating, by the neural network and in accordance with the values of the set of neural network parameters, the action selection output based on the combined embedding comprises:

processing the combined embedding using an action selection neural network of the neural network to generate the action selection output.

20. The non-transitory computer storage media of claim 18 , wherein the language encoder model is a recurrent neural network.

Assignments (2)

Continuity (4)

References Cited (80)