GENERATING MULTIMODAL DATA USING A MEDIA ITEM GENERATION NEURAL NETWORK AND A TOKEN GENERATION NEURAL NETWORK

1 . A method performed by one or more computers and for generating multimodal data using a system comprising a token generation neural network and a media generation neural network, the method comprising:

obtaining a multimodal input that comprises a set of one or more media items and one or more prompt tokens of a different modality;

processing each of the one or more media items using a first media encoder neural network to generate a respective set of media tokens for each of the one or more media items;

generating an input sequence of multimodal tokens that comprises the prompt tokens and the media tokens for the one or more media items;

processing the input sequence of multimodal tokens using the token generation neural network to generate an output sequence of multimodal tokens, comprising:

determining that a criterion is satisfied as of a given multimodal token in the output sequence; and

in response:

processing each of the one or more media items using a second media encoder neural network to generate a respective encoded representation for each of the one or more media items; and

generating an output media item using the media generation neural network conditioned on (i) features representing a current output sequence of multimodal tokens as of the given multimodal token obtained from the token generation neural network and (ii) the encoded representations of the one or more media items.

2 . The method of claim 1 , wherein generating the output sequence of multimodal tokens comprises autoregressively, for each successive position in the output sequence of multimodal tokens:

processing a combined sequence comprising the input sequence of multimodal tokens and a current output sequence of multimodal tokens, using the token generation neural network, to generate a next multimodal token for the output sequence of multimodal tokens, and

appending the next multimodal token to the current output sequence of multimodal tokens.

3 . The method of claim 1 , wherein determining that a criterion is satisfied as of a given multimodal token in the output sequence comprises determining that the next multimodal token is a start-of-media token.

4 . The method of claim 1 , wherein the first media encoder neural network is different from the second media encoder neural network.

5 . The method of claim 1 , wherein the first media encoder neural network is the second media encoder neural network.

6 . The method of claim 1 , wherein processing the input sequence of multimodal tokens using the token generation neural network to generate an output sequence of multimodal tokens further comprises:

processing the output media item to generate a sequence of media tokens; and

appending the sequence of media tokens to the current output sequence of multimodal tokens as the next multimodal tokens in the output sequence of multimodal tokens after the given multimodal token.

7 . The method of claim 6 , wherein processing the input sequence of multimodal tokens using the token generation neural network to generate an output sequence of multimodal tokens further comprises:

continuing to generate, using the token generation neural network, further multimodal tokens after appending the sequence of media tokens to the current output sequence of multimodal tokens.

8 . The method of claim 7 , wherein the token generation neural network comprises one or more self-attention neural network layers, the method further comprising applying a causal mask to the self-attention neural network layers whilst the self-attention neural network layers are processing the multimodal tokens, and using bi-directional attention whilst the self-attention neural network layers are processing the media tokens.

9 . The method of claim 6 , wherein processing the output media item to generate a sequence of media tokens comprises processing the output media item using the first media encoder neural network to generate the sequence of media tokens.

10 . The method of claim 1 , wherein the one or more prompt tokens comprise tokens representing one or more of text or audio data.

11 . The method of claim 1 , wherein the output sequence of multimodal tokens comprises multimodal tokens representing text or audio data elements.

12 . The method of claim 1 , wherein the media generation neural network is a diffusion neural network, and wherein generating the output media item comprises:

initializing the media item or a latent representation thereof, by sampling values for elements of the media item or for the latent representation from a noise distribution; and

at each of a series of time steps: determining an updated version of the media item or the latent representation thereof, by processing data specifying the time step, and the media item or the latent representation thereof, at the time step, using the media generation neural network conditioned on (i) the features representing a current output sequence of multimodal tokens as of the given multimodal token obtained from the token generation neural network and (ii) the encoded representations of the one or more media items, to determine a reduced noise version of the media item or of the latent representation thereof.

13 . The method of claim 1 , wherein the features representing the current output sequence of multimodal tokens as of the given multimodal token obtained from the token generation neural network comprise:

i) respective output embeddings for at least a subset of the multimodal tokens in a current input sequence generated by a last attention layer block of the generative neural network by processing the current input sequence using the generative neural network, wherein the current input sequence comprises the current output sequence;

ii) a respective output embedding for a predetermined additional token generated by the last attention layer block by processing an updated sequence that includes the predetermined additional token appended to the current input sequence using the token generation neural network; or

iii) both.

14 . The method of claim 1 , wherein the media generation neural network has been trained on a task that requires generating output media items conditioned on features generated by the token generation neural network while holding the token generation neural network fixed.

15 . The method of claim 14 , wherein the media generation neural network has been pre-trained on a different media generation task prior to the training on the task that requires generating output media items conditioned on features generated by the token generation neural network.

16 . The method of claim 14 , wherein the second media encoder neural network is trained jointly with the media generation neural network on the task that requires generating output media items conditioned on features generated by the token generation neural network while holding the token generation neural network fixed.

17 . The method of claim 14 , wherein the second media encoder neural network is held fixed while the media generation neural network is trained on the task that requires generating output media items conditioned on features generated by the token generation neural network while holding the token generation neural network fixed.

18 . The method of claim 1 , wherein the input sequence of multimodal tokens further comprises media tokens representing one or more additional media items and wherein the media tokens for the set of one or more media items follow the media tokens representing the one or more additional tokens in the sequence.

19 . The method of claim 1 , wherein the one or more prompt tokens specify one or more edits to be applied to the one or more media items to generate the output media item.

20 . A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to generate an output media item using a token generation neural network and a media generation neural network, the operations comprising:

obtaining a multimodal input that comprises a set of one or more media items and one or more prompt tokens of a different modality;

processing each of the one or more media items using a first media encoder neural network to generate a respective set of media tokens for each of the one or more media items;

generating an input sequence of multimodal tokens that comprises the prompt tokens and the media tokens for the one or more media items;

processing the input sequence of multimodal tokens using the token generation neural network to generate an output sequence of multimodal tokens, comprising:

determining that a criterion is satisfied as of a given multimodal token in the output sequence; and

in response:

processing each of the one or more media items using a second media encoder neural network to generate a respective encoded representation for each of the one or more media items; and

generating an output media item using the media generation neural network conditioned on (i) features representing a current output sequence of multimodal tokens as of the given multimodal token obtained from the token generation neural network and (ii) the encoded representations of the one or more media items.

21 . One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to generate an output media item using a token generation neural network and a media generation neural network, the operations comprising:

obtaining a multimodal input that comprises a set of one or more media items and one or more prompt tokens of a different modality;

processing each of the one or more media items using a first media encoder neural network to generate a respective set of media tokens for each of the one or more media items;

generating an input sequence of multimodal tokens that comprises the prompt tokens and the media tokens for the one or more media items;

processing the input sequence of multimodal tokens using the token generation neural network to generate an output sequence of multimodal tokens, comprising:

determining that a criterion is satisfied as of a given multimodal token in the output sequence; and

in response:

processing each of the one or more media items using a second media encoder neural network to generate a respective encoded representation for each of the one or more media items; and

generating an output media item using the media generation neural network conditioned on (i) features representing a current output sequence of multimodal tokens as of the given multimodal token obtained from the token generation neural network and (ii) the encoded representations of the one or more media items.