Controlling interactive agents using multi-modal inputs
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling agents. In particular, an interactive agent can be controlled based on multi-modal inputs that include both an observation image and a natural language text sequence.
1 . A method performed by one or more computers and for controlling an agent interacting with an environment, the method comprising, at each of a plurality of time steps:
receiving an observation image characterizing a state of the environment at the time step;
receiving a natural language text sequence for the time step that characterizes a task being performed by the agent in the environment at the time step;
processing the observation image using an image embedding neural network to generate a plurality of image embeddings that represent the observation image;
processing the natural language text sequence using a text embedding neural network to generate a plurality of text embeddings that represent the natural language text sequence;
processing an input comprising the image embeddings, the text embeddings, and a set of one or more dedicated embeddings using a multi-modal Transformer neural network to generate an aggregated embedding, wherein the multi-modal Transformer neural network is configured to (i) apply self-attention over at least the text embeddings and the image embeddings to generate respective updated embeddings for at least the plurality of dedicated embeddings and (ii) generate the aggregated embedding from at least the respective updated embeddings for the dedicated embeddings;
selecting, using the aggregated embedding, one or more actions to be performed by the agent in response to the observation image; and
causing the agent to perform the one or more selected actions.
2 . The method of claim 1 , wherein the multi-modal Transformer neural network comprises one or more self-attention layers that each have one or more self-attention heads, and wherein applying self-attention comprises processing the input through the one or more self-attention layers.
3 . The method of claim 2 , wherein the set of one or more dedicated embeddings are the same for each of the plurality of time steps and are not dependent on the observation images at the plurality of time steps.
4 . The method of claim 3 , wherein applying self-attention comprises generating respective updated embeddings for the text embeddings and the dedicated embeddings without updating the image embeddings.
5 . The method of claim 3 , wherein each self-attention head of each self-attention layer is configured to:
receive a head input comprising (i) the image embeddings generated by the image embedding neural network and (ii) respective current embeddings for the text embeddings and the dedicated embeddings;
generate, from the respective current embeddings, a respective query corresponding to each text embedding and each dedicated embedding;
generate, from the image embeddings and the respective current embeddings, a respective key corresponding to each image embedding, each text embedding, and each dedicated embedding;
generate, from the image embeddings and the respective current embeddings, a respective value corresponding to each image embedding, each text embedding, and each dedicated embedding; and
apply query-key-value attention over the respective queries, keys, and values to generate a respective initial updated embedding for each text embedding and each dedicated embedding without updating the image embeddings.
6 . The method of claim 3 , wherein generating the aggregated embedding comprises:
aggregating the respective updated embeddings for the text embeddings and the dedicated embeddings to generate an initial aggregated embedding; and
combining the respective updated embeddings for the dedicated embeddings with the initial aggregated embedding to generate the aggregated embedding.
7 . The method of claim 6 , wherein the combining comprises concatenating each respective updated embedding for each dedicated embedding and the initial aggregated embedding.
8 . The method of claim 1 , wherein selecting, using the aggregated embedding, one or more actions to be performed by the agent in response to the observation image comprises:
generating a state representation from the aggregated embedding; and
selecting the one or more actions using the state representation.
9 . The method of claim 8 , wherein generating the state representation comprises processing the aggregated embedding using a memory neural network.
10 . The method of claim 9 , wherein the memory neural network is a recurrent neural network.
11 . The method of claim 8 , further comprising:
processing the state representation using a natural language generation neural network to generate an output text sequence for the time step.
12 . The method of claim 11 , wherein the natural language text sequence is generated by transcribing a verbalized utterance from another agent in the environment, and wherein the method further comprises:
generating speech representing the output text sequence for the time step; and
causing the agent to verbalize the generated speech.
13 . The method of claim 11 , further comprising:
processing the state representation using a text no-op neural network to generate an indication of whether text should be generated at the time step; and
wherein processing the state representation for the time step using a natural language generation neural network to generate an output text sequence for the time step comprises:
only generating the output text sequence when the indication indicates that text should be generated at the time step.
14 . The method of claim 8 , wherein selecting the one or more actions using the state representation comprises:
processing the state representation using an action policy neural network to select a single action to be performed in response to the image observation.
15 . The method of claim 8 , wherein selecting the one or more actions using the state representation comprises:
processing the state representation to select a sequence of a plurality of actions to be performed in response to the image observation, the sequence comprising a respective action at each of a plurality of positions.
16 . The method of claim 15 , wherein processing the state representation comprises:
processing the state representation using a high-level controller neural network to generate a respective low-level input for each position in the sequence; and
for each position, processing the respective low-level input for the position using a policy neural network to select the action to be performed by the agent at the position in the sequence.
17 . The method of claim 16 , wherein the high-level controller neural network auto-regressively generates the respective low-level inputs for each position in the sequence after receiving as input the state representation.
18 . The method of claim 17 , wherein the high-level controller neural network is a recurrent neural network.
19 . The method of claim 8 , further comprising:
processing the state representation using an action no-op neural network to generate an indication of whether any actions should be performed at the time step; and
wherein causing the agent to perform the one or more actions comprises:
only causing the agent to perform the actions when the indication indicates that actions should be performed at the time step.
20 . The method of claim 1 , wherein the natural language text sequence is generated by transcribing a verbalized utterance from another agent in the environment.
21 . A method performed by one or more computers and for controlling an agent interacting with an environment, the method comprising, at each of a plurality of time steps:
receiving an observation image for the time step characterizing a state of the environment at the time step;
receiving a natural language text sequence for the time step that characterizes a task being performed by the agent in the environment at the time step;
processing the observation image and the natural language text sequence to generate a state representation for the time step;
generating a sequence of a plurality of actions to be performed by the agent in response to the observation image at the time step, the sequence comprising a respective action to be performed by the agent at each of a plurality of positions, the generating comprising:
processing the state representation using a high-level controller neural network to generate a respective low-level input for each position in the sequence; and
for each position, processing the respective low-level input for the position using an action policy neural network to generate the action to be performed by the agent at the position in the sequence; and
causing the agent to perform the sequence of actions.
22 . The method of claim 21 , further comprising:
processing the state representation for the time step using a natural language generation neural network to generate an output text sequence for the time step.
23 . The method of claim 22 , wherein the natural language text sequence is generated by transcribing a verbalized utterance from another agent in the environment, and wherein the method further comprises:
generating speech representing the output text sequence for the time step; and
causing the agent to verbalize the generated speech.
24 . The method of claim 22 , further comprising:
processing the state representation using a text no-op neural network to generate an indication of whether text should be generated at the time step; and
wherein processing the state representation for the time step using a natural language generation neural network to generate an output text sequence for the time step comprises:
only generating the output text sequence when the indication indicates that text should be generated at the time step.
25 . The method of claim 21 , further comprising:
processing the state representation using an action no-op neural network to generate an indication of whether any actions should be performed at the time step; and
wherein causing the agent to perform the sequence of actions comprises:
only causing the agent to perform the sequence of actions when the indication indicates that actions should be performed at the time step.
26 . The method of claim 21 , wherein the high-level controller neural network auto-regressively generates the respective low-level inputs for each position in the sequence after receiving as input the state representation.
27 . The method of claim 21 , wherein the action policy neural network comprises a respective sub-network for each of a plurality of sub-actions, and wherein processing the respective low-level input for the position using an action policy neural network to generate the action to be performed by the agent at the position in the sequence comprises:
for each of the plurality of sub-actions, processing an input comprising the respective low-level input for the position using the sub-network for the sub-action to select a value for the sub-action for the position.
28 . The method of claim 27 , wherein, for at least one of the sub-actions, the input comprises the value selected for one or more of the other sub-actions at the position.
29 . The method of claim 21 , wherein processing the observation image and the natural language text sequence to generate a state representation for the time step comprises processing the observation image and the natural language text sequence using a perceptual encoder neural network that is configured to receive as input an observation image characterizing a state of an environment at a time step and a natural language text sequence characterizing the environment and to generate as output an encoded representation for the time step, and wherein the perceptual encoder has been trained by performing operations comprising:
obtaining a plurality of training pairs, the plurality of training pairs comprising:
(i) a first pair comprising an observation image and a natural language text sequence that both correspond to the same time step, and
(ii) one or more second pairs each comprising an observation image and a natural language text sequence that correspond to different time steps;
processing each training pair using the perceptual encoder neural network to generate a respective encoded representation for each training pair;
processing the respective encoded representation for each training pair using a discriminator neural network that is configured to process the respective encoded representations to generate, for each training pair, a respective discriminator score that represents a likelihood that the observation image and the natural language text sequence in the pair from which the encoded representation was generated correspond to the same time step; and
training the perceptual encoder neural network using a contrastive learning loss based on the respective discriminator scores for each training pair.
30 . The method of claim 29 , wherein the contrastive learning loss encourages the respective discriminator score for the first pair to be higher while encouraging the respective discriminator scores for the second pairs to be lower.
31 . The method of claim 29 , wherein the discriminator neural network is a feedforward neural network that processes each encoded representation independently to generate the respective discriminator score for the corresponding training pair.
32 . The method of claim 29 , further comprising:
generating, from the encoded representation for the first training pair and using at least a policy neural network, a respective probability distribution over a set of actions for each of one or more positions in an action sequence;
obtaining data specifying a respective ground truth action performed by an expert agent at each of the one or more positions; and
training at least the policy neural network and the perceptual encoder neural network based on a behavior cloning loss that measures, for each position in the action sequence, the probability assigned to the ground truth action at the position by the probability distribution for the position.
33 . The method of claim 29 , further comprising:
training the discriminator neural network on an objective function that measures, for the first training pair, an error between the respective discriminator score for the first training pair and a first score that indicates that the observation image and the text sequence in the pair correspond to the same time step and, for each second training pair, an error between the respective discriminator score for the second training pair and a second score that indicates that the observation image and the text sequence in the pair do not correspond to the same time step.