IP Library › Granted Patent US 12,469,186
Granted Patent B2
US 12,469,186 · App. 19/193,787 · Granted Nov 11, 2025

Systems and methods for generating multimodal data using a single-tower architecture with a data generation subsystem

Inventors: Mostafa Dehghani (Amsterdam, NL); Phillip Lippe (Amsterdam, NL); Emiel Hoogeboom (Amsterdam, NL); Jonathan Heek (Hilversum, NL)
Assignee: GDM Holding LLC
G06T11/00G06F40/284G06F40/40G06V10/774G06V10/82G06V30/19147G10L25/18G10L25/30
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,469,186
App. No.
19/193,787
Granted
Nov 11, 2025
Kind
B2
Abstract

A computer-implemented method of generating multimodal data. The method comprises using a token generation neural network to generate, autoregressively, an output sequence of multimodal tokens, and in response to a next multimodal token being a start-of-image token, generating an image using an image generation subsystem conditioned on features representing the current sequence of multimodal tokens obtained from the token generation neural network. The method further comprises processing the image to convert pixels of the image into a sequence of image tokens, each image token comprising a block encoding of values of the pixels in a different region of the image that maps a set of values of the pixels to a respective image token, and appending the sequence of image tokens to the current output sequence of multimodal tokens as the next multimodal tokens in the output sequence of multimodal tokens.

Claims (83)

1 . A computer-implemented method of generating multimodal data using a system comprising a token generation neural network, and an image generation subsystem comprising an image generation neural network, the method comprising:

receiving a prompt sequence that defines an input sequence of multimodal tokens, and processing the input sequence of multimodal tokens using the token generation neural network to generate an output sequence of multimodal tokens, wherein a multimodal token represents a data element of one of a plurality of modalities;

wherein generating the output sequence of multimodal tokens comprises autoregressively, for each successive position in the output sequence of multimodal tokens:

processing a combined sequence comprising the input sequence of multimodal tokens and a current output sequence of multimodal tokens, using the token generation neural network, to generate a next multimodal token for the output sequence of multimodal tokens, and appending the next multimodal token to the current output sequence of multimodal tokens;

the method further comprising, in response to the next multimodal token being a start-of-image token:

generating an image using the image generation subsystem conditioned on features representing the current output sequence of multimodal tokens obtained from the token generation neural network;

processing the image to convert pixels of the image into a sequence of image tokens, each image token comprising a block encoding of values of the pixels in a different region of the image that maps a set of values of the pixels to a respective image token; and

appending the sequence of image tokens to the current output sequence of multimodal tokens as the next multimodal tokens in the output sequence of multimodal tokens.

2 . The method of claim 1 , further comprising continuing to process the combined sequence after appending the sequence of image tokens to the current output sequence of multimodal tokens, to generate further multimodal tokens for the output sequence of multimodal tokens.

3 . The method of claim 1 , wherein the token generation neural network comprises one or more self-attention neural network layers, the method further comprising using bi-directional attention whilst the self-attention neural network layers are processing the image tokens.

4 . The method of claim 1 , wherein the input sequence of multimodal tokens comprises multimodal tokens representing text, audio, or image data elements, and wherein the output sequence of multimodal tokens comprises multimodal tokens representing text or audio data elements.

5 . The method of claim 1 , wherein processing the image to convert pixels of the image into a sequence of image tokens, comprises processing the image to divide the image into a set of regions that tile the image, and determining each image token as a linear projection of the values of the pixels in each respective region.

6 . The method of claim 1 , wherein the image generation subsystem is a diffusion model image generation subsystem, and wherein generating the image comprises:

initializing the image or a latent vector representation thereof, by sampling values for the pixels of the image or for the latent vector representation from a noise distribution; and

at each of a series of time steps: determining an updated version of the image or the latent vector representation thereof, by processing the time step, and the image or the latent vector representation thereof, at the time step, using the image generation neural network conditioned on the features representing the current output sequence of multimodal tokens, to determine a reduced noise version of the image or of the latent vector representation thereof.

7 . The method of claim 1 , wherein the image generation neural network has a U-net architecture comprising one or more cross-attention neural network layers, and wherein using the image generation subsystem to generate the predicted image conditioned on features representing the current output sequence, comprises:

using the one or more cross-attention neural network layers to attend to features of the current output sequence obtained by processing the current output sequence using the token generation neural network.

8 . The method of claim 1 , comprising:

processing the combined sequence including the start-of-image token using the token generation neural network to generate features of a summary multimodal token; and

wherein generating an image using the image generation subsystem conditioned on features representing the current output sequence of multimodal tokens obtained from the token generation neural network comprises generating the image using the image generation subsystem conditioned on the features of the summary multimodal token.

9 . The method of claim 1 , wherein the image comprises an audio spectrogram; the method further comprising converting the audio spectrogram to time series audio data for an audio waveform.

10 . The method of claim 9 , wherein the prompt sequence comprises text or audio that defines an audio generation task; and wherein the time series audio data for the audio waveform defines audio that is specified by the prompt sequence.

11 . The method of claim 1 , wherein the prompt sequence comprises text or audio that defines an image generation task or image processing task; and wherein the image defines a result of the task; in particular wherein:

i) the task comprises generating an image specified by the prompt;

ii) the prompt includes an image and the task comprises generating a modified version of the image, where a modification to be performed is described by the prompt;

iii) the prompt includes an image and the task is an optical character recognition task that involves generating an output sequence of multimodal tokens that represents words or characters in the image;

iv) the prompt includes an image and the task comprises generating an output sequence of multimodal tokens that represents an answer to a question about the image;

v) the prompt includes an image and identifies one or more objects in the image and the task comprises generating an output sequence of multimodal tokens that defines a presence, location, orientation, or count of one or more of the objects in the image;

vi) the prompt includes an image and the task comprises generating an output sequence of multimodal tokens that describes a content of the image or that classifies a content of the image into one or more of a plurality of categories;

vii) the prompt includes an image and defines a goal for a mechanical agent acting in a real world environment and the task comprises generating an output sequence of multimodal tokens that defines one or more actions to be performed by the mechanical agent to achieve the goal.

12 . A computer-implemented method of training a system for generating multimodal data items, the system comprising a token generation neural network, an image generation subsystem comprising an image generation neural network, and a block encoder, the method comprising:

obtaining a set of multimodal training examples, each training example comprising text or audio data and at least one training image and, for each training example:

generating a corresponding sequence of ground truth multimodal tokens representing the text or audio data and the training image, wherein generating the ground truth multimodal tokens representing the training image comprises processing the training image using the block encoder to convert pixels of the training image into a sequence of image tokens, each image token comprising a block encoding of values of the pixels in a different region of the training image that maps a set of values of the pixels to a respective image token; and

wherein the sequence of ground truth multimodal tokens includes a start-of-image token before a ground truth multimodal token representing the training image; and

determining a plurality of training sequences each comprising a subsequence of the sequence of ground truth multimodal tokens; and for each training sequence:

in response to determining that a last ground truth multimodal token in the subsequence is the start-of-image token, using the image generation subsystem to generate a predicted image conditioned on features representing the subsequence, and

processing the subsequence using the token generation neural network to generate a predicted next multimodal token when the last ground truth multimodal token in the subsequence is not the start-of-image token; and

training the system based on the training examples using i) an image reconstruction objective that depends on the predicted image and the training image, and ii) a token prediction objective that depends on the predicted next multimodal token and a next ground truth multimodal token after the subsequence, the training including updating learnable parameters of the token generation neural network, and the image generation neural network.

13 . The method of claim 12 , wherein the multimodal training examples include training examples in which the at least one training image is after the text or audio data and training examples in which the at least one training image is before the text or audio data.

14 . The method of claim 12 , wherein the training includes updating learnable parameters of the block encoder, and wherein said training includes back propagating gradients of the image reconstruction objective and of the token prediction objective through the token generation neural network into the block encoder to update the learnable parameters of the block encoder.

15 . The method of claim 12 , wherein the block encoder comprises a linear image region encoding neural network, the method comprising linearly projecting the set of values of the pixels of each region of the training image to a respective image token using the linear image region encoding neural network; and wherein the training comprises back propagating the gradients into the linear image region encoding neural network to update weights of the linear image region encoding neural network.

16 . The method of claim 12 , wherein using the image generation subsystem to generate a predicted image conditioned on features representing the subsequence, comprises:

processing the training sequence including the start-of-image token using the token generation neural network to generate features of a summary multimodal token; and

using the image generation subsystem to generate the predicted image conditioned on the features of the summary multimodal token.

17 . The method of claim 12 , wherein the image generation subsystem is a diffusion model image generation subsystem, wherein using the image generation subsystem to generate a predicted image comprises:

sampling a time step from a distribution;

generating a noisy version of the training image by adding noise to the training image at a level determined by the time step, the added noise defining a noise image added to the training image; and

processing the noisy version of the training image and the time step using the image generation neural network to generate the predicted image; and

wherein the predicted image represents either the noise image or a reconstructed version of the training image, and wherein the image reconstruction objective depends on, respectively, either a difference between the predicted image and the noise image, or a difference between the predicted image and the training image.

18 . The method of claim 12 , wherein the image generation neural network has a U-net architecture comprising one or more cross-attention neural network layers, and wherein using the image generation subsystem to generate the predicted image conditioned on features representing the subsequence, comprises:

using the one or more cross-attention neural network layers to attend to features of the subsequence obtained by processing the subsequence using the token generation neural network.

19 . The method of claim 12 , further comprising:

replacing the image generation subsystem comprising the image generation neural network with a second image generating subsystem comprising a second, larger image generation neural network;

freezing the learnable parameters of the token generation neural network and the block encoder; and

training the second, larger image generation neural network using further multimodal training examples, each further training example comprising text or audio data and at least one training image, to update learnable parameters of the second, larger image generation neural network.

20 . The method of claim 12 , wherein generating the corresponding sequence of ground truth multimodal tokens comprises:

inserting the start-of-image token in the sequence before the ground truth multimodal token representing the training image.

21 . The method of claim 12 ,

wherein obtaining the set of multimodal training examples comprises obtaining training examples comprising i) text or first audio data and ii) second audio data, wherein the text or first audio data, and the second audio data, are semantically related; the method further comprising:

converting the second audio data of each training example into an audio spectrogram to obtain the training image for the training example.

22 . The method of claim 21 , further comprising using the system, after the training, to perform an audio generation task, the method comprising:

using the system to process a prompt sequence that defines an input sequence of multimodal tokens for the system, wherein the prompt sequence comprises text or audio that defines the audio generation task; and wherein the time series audio data for the audio waveform defines audio that is specified by the prompt sequence.

23 . The method of claim 12 , further comprising using the system, after the training, to perform an image generation task or image processing task, the method comprising:

using the system to process a prompt sequence that defines an input sequence of multimodal tokens for the system, wherein the prompt sequence comprises text or audio that defines an image generation task or image processing task, and wherein the image defines a result of the task; in particular wherein:

i) the task comprises generating an image specified by the prompt;

ii) the prompt includes an image and the task comprises generating a modified version of the image, where a modification to be performed is described by the prompt;

iii) the prompt includes an image and the task is an optical character recognition task that involves generating an output sequence of multimodal tokens that represents words or characters in the image;

iv) the prompt includes an image and the task comprises generating an output sequence of multimodal tokens that represents an answer to a question about the image;

v) the prompt includes an image and identifies one or more objects in the image and the task comprises generating an output sequence of multimodal tokens that defines a presence, location, orientation, or count of one or more of the objects in the image;

vi) the prompt includes an image and the task comprises generating an output sequence of multimodal tokens that describes a content of the image or that classifies a content of the image into one or more of a plurality of categories;

vii) the prompt includes an image and defines a goal for a mechanical agent acting in a real world environment and the task comprises generating an output sequence of multimodal tokens that defines one or more actions to be performed by the mechanical agent to achieve the goal.

24 . The method of claim 23 , performed prior to further training to perform the image generation task or image processing task.

25 . A system comprising:

one or more computers; and

one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for generating multimodal data using a neural network system comprising a token generation neural network, and an image generation subsystem comprising an image generation neural network, the operations comprising:

receiving a prompt sequence that defines an input sequence of multimodal tokens, and processing the input sequence of multimodal tokens using the token generation neural network to generate an output sequence of multimodal tokens, wherein a multimodal token represents a data element of one of a plurality of modalities;

wherein generating the output sequence of multimodal tokens comprises autoregressively, for each successive position in the output sequence of multimodal tokens:

processing a combined sequence comprising the input sequence of multimodal tokens and a current output sequence of multimodal tokens, using the token generation neural network, to generate a next multimodal token for the output sequence of multimodal tokens, and

appending the next multimodal token to the current output sequence of multimodal tokens;

the method further comprising, in response to the next multimodal token being a start-of-image token:

generating an image using the image generation subsystem conditioned on features representing the current output sequence of multimodal tokens obtained from the token generation neural network;

processing the image to convert pixels of the image into a sequence of image tokens, each image token comprising a block encoding of values of the pixels in a different region of the image that maps a set of values of the pixels to a respective image token; and

appending the sequence of image tokens to the current output sequence of multimodal tokens as the next multimodal tokens in the output sequence of multimodal tokens.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 1, 2025
From: DEHGHANI, MOSTAFA; LIPPE, PHILLIP; HOOGEBOOM, EMIEL; HEEK, JONATHAN
To: GDM HOLDING LLC
Reel/Frame 071583/0955 →
Continuity (2)
Provisional Application 63640140 · Apr 29, 2024
Related Publication 20250336101A1 · Oct 30, 2025
References Cited (100)
US 20230237772A1 · Li · 2023 [cited by examiner]
US 20230262293A1 · Barbieri · 2023 [cited by examiner]
US 20230419551A1 · Harikumar · 2023 [cited by examiner]
US 20240029411A1 · Edraki · 2024 [cited by examiner]
US 20240311652A1 · Kulkarni · 2024 [cited by examiner]
US 20240320529A1 · Dathathri · 2024 [cited by examiner]
Achiam et al., “GPT-4 Technical Report,” CoRR, Mar. 15, 2023, arxiv.org/abs/2303.08774, 100 pages. [cited by applicant]
Adept.ai [online], “Adept Fuyu-Heavy: A new multimodal model,” Jan. 24, 2024, retrieved on May 5, 2025, retrieved from URL <https://www.adept.ai/blog/adept-fuyu-heavy>, 8 pages. [cited by applicant]
Adept.ai [online], “Fuyu-8B: A Multimodal Architecture for AI Agents,” Oct. 17, 2023, retrieved on May 9, 2025, retrieved from URL <https://www.adept.ai/blog/fuyu-8b>, 17 pages. [cited by applicant]
Aghajanyan et al., “CM3: A Causal Masked Multimodal Model of the Internet,” CoRR, Jan. 19, 2022, arXiv:2201.07520, 20 pages. [cited by applicant]
Aghajanyan et al., “Scaling laws for generative mixed-modal language models,” CoRR, Jan. 10, 2023, arXiv:2301.03728, 19 pages. [cited by applicant]
Ahuja et al., “Megaverse: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks,” CoRR, Nov. 13, 2023, arXiv:2311.07463, 40 pages. [cited by applicant]
Aiello et al., “Jointly Training Large Autoregressive Multimodal Models,” CoRR, Sep. 27, 2023, arXiv:2309.15564, 21 pages. [cited by applicant]
Alabdulmohsin et al., “Fractal Patterns May Unravel the Intelligence in Next-Token Prediction,” CoRR, Feb. 2, 2024, arXiv:2402.01825, 17 pages. [cited by applicant]
Alayrac et al., “Flamingo: a visual language model for few-shot learning,” Advances in Neural Information Processing Systems 35 (NeurIPS 2022) Main Conference Track, 2022, 21 pages. [cited by applicant]
An et al., “Open-LEAF: Open-Domain Interleaved Image-Text Generation and Evaluation,” CoRR, Oct. 11, 2023, arXiv:2310.07749, 19 pages. [cited by applicant]
Anil et al., “Gemini: A Family of Highly Capable Multimodal Models,” CoRR, Dec. 19, 2023, arxiv.org/abs/2312.11805, 90 pages. [cited by applicant]
Anil et al., “Palm 2 technical report,” CoRR, May 17, 2023, arXiv:2305.10403, 93 pages. [cited by applicant]
Bao et al., “VL-BEiT: Generative vision-language pretraining,” CoRR, Jun. 2, 2022, arXiv:2206.01127, 12 pages. [cited by applicant]
Bar-Tal et al., “Lumiere: A Space-Time Diffusion Model for Video Generation,” CoRR, Jan. 23, 2024, arXiv:2401.12945, 20 pages. [cited by applicant]
Betker et al., “Improving Image Generation with Better Captions,” Computer Science, OpenAI Papers, 2(3):8. [cited by applicant]
Brooks et al., “InstructPix2Pix: Learning to follow image editing instructions,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 18392-18402. [cited by applicant]
Brown et al., “Language Models are Few-Shot Learners,” Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020, 25 pages. [cited by applicant]
Chen et al., “Generative Pretraining From Pixels,” Proceedings of the 37th International Conference on Machine Learning, 2020, 119:1691-1703. [cited by applicant]
Chen et al., “PaLI: A Jointly-Scaled Multilingual Language-Image Model,” The Eleventh International Conference on Learning Representations, Feb. 1, 2023, 33 pages. [cited by applicant]
Chen et al., “UNITER: Universal image-text representation learning,” Computer Vision—ECCV 2020, Sep. 24, 2020, pp. 104-102. [cited by applicant]
Chen, et al., “PaLI-X: On Scaling up a Multilingual Vision and Language Model,” CORR, May 29, 2023, arXiv:2305.18565v1, 30 pages. [cited by applicant]
Conneau et al., “Unsupervised Cross-lingual Representation Learning at Scale,” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Jul. 2020, pp. 8440-8451. [cited by applicant]
Dai et al., “InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning,” CoRR, May 11, 2023, arXiv:2305.06500, 17 pages. [cited by applicant]
Dehghani et al., “Scaling Vision Transformers to 22 Billion Parameters,” Proceedings of the 40th International Conference on Machine Learning, 2023, 202:7480-7512. [cited by applicant]
Deng et al., “ImageNet: A large-scale hierarchical image database,” 2009 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 20-25, 2009, pp. 248-255. [cited by applicant]
Ding et al., “CogView: Mastering Text-to-Image Generation via Transformers,” Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 14 pages. [cited by applicant]
Dosovitskiy et al., “An Image is Worth 16x16Words: Transformers for Image Recognition at Scale,” International Conference on Learning Representations, Jan. 12, 2021, 21 pages. [cited by applicant]
Ebert et al., “Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets,” CoRR, Sep. 27, 2021, arXiv:2109.13396, 7 pages. [cited by applicant]
El-Nouby et al., “Scalable Pre-training of Large Autoregressive Image Models,” CoRR, Jan. 16, 2024, arXiv:2401.08541, 14 pages. [cited by applicant]
Esser et al., “ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis,” Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 15 pages. [cited by applicant]
Esser et al., “Taming transformers for high-resolution image synthesis,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 12873-12883. [cited by applicant]
Gadre et al., “DataComp: In search of the next generation of multimodal datasets,” 37th Conference on Neural Information Processing Systems (NeurIPS 2023), Track on Datasets and Benchmark, 2023, 21 pages. [cited by applicant]
Gafni et al., “Make-A-Scene: Scene-based text-to-image generation with human priors,” European Conference on Computer Vision, Oct. 31, 2022, pp. 89-106. [cited by applicant]
Gemmeke et al., “Audio set: An ontology and human-labeled dataset for audio events,” 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Mar. 5-9, 2017, pp. 776-780. [cited by applicant]
Goyal et al., “Larger-Scale Transformers for Multilingual Masked Language Modeling,” Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), Aug. 2021, pp. 29-33. [cited by applicant]
Gu et al., “Vector Quantized Diffusion Model for Text-to-Image Synthesis,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2022, pp. 10696-10706. [cited by applicant]
Hashimoto, “Model Performance Scaling with Multiple Data Sources,” Proceedings of the 38th International Conference on Machine Learning, Jul. 18-24, 2021, 139:4107-4116. [cited by applicant]
He et al., “Deep residual learning for image recognition,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770-778. [cited by applicant]
Ho et al., “Classifier-Free Diffusion Guidance,” CoRR, Jul. 26, 2022, arXiv:2207.12598, 14 pages. [cited by applicant]
Ho et al., “Denoising Diffusion Probabilistic Models,” CoRR, Jun. 19, 2020, arXiv:2006:11239, 25 pages. [cited by applicant]
Hoffmann et al., “Training compute-optimal large language models,” CoRR, Mar. 29, 2022, arXiv:2203.15556, 36 pages. [cited by applicant]
Hoogeboom et al., “Simple Diffusion: End-to-End Diffusion for High Resolution Images,” Proceedings of the 40th International Conference on Machine Learning, 2023, 202:13213-13232. [cited by applicant]
Huang et al., “Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably),” Proceedings of the 39th International Conference on Machine Learning, Jul. 17-23, 162:9226-9259. [cited by applicant]
Huang et al., “Not All Image Regions Matter: Masked Vector Quantization for Autoregressive Image Generation,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 2002-2011. [cited by applicant]
Huang et al., “Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, p… [cited by applicant]
Jain et al., “A Data-Based Perspective on Transfer Learning,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2023, pp. 3613-3622. [cited by applicant]
Jia et al., “Scaling up visual and vision-language representation learning with noisy text supervision,” Proceedings of the 38th International Conference on Machine Learning, 2021, 139:4904-4916. [cited by applicant]
Jiang et al., “Mixtral of experts,” CoRR, Jan. 8, 2024, arXiv:2401.04088, 13 pages. [cited by applicant]
Kaplan et al., “Scaling laws for neural language models,” CoRR, Jan. 23, 2020, arXiv:2001.08361, 30 pages. [cited by applicant]
Karras et al., “Analyzing and Improving the Training Dynamics of Diffusion Models,” CoRR, Dec. 5, 2023, arXiv:2312.02696, 39 pages. [cited by applicant]
Kay et al., “The Kinetics Human Action Video Dataset,” CoRR, May 19, 2017, arXiv:1705.06950, 22 pages. [cited by applicant]
Kim et al., “ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision,” Proceedings of the 38th International Conference on Machine Learning, 2021, 139:5583-5594. [cited by applicant]
Krishna et al., “Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations,” CoRR, Feb. 23, 2016, arXiv:1602.07332, 44 pages. [cited by applicant]
Kuznetsova et al., “The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale,” CoRR, Nov. 2, 2018, arXiv:1811.00982, 26 pages. [cited by applicant]
Laurençon et al., “OBELICS: An open web-scale filtered dataset of interleaved image-text documents,” CoRR, Jun. 21, 2023, arXiv:2306.16527, 51 pages. [cited by applicant]
Li et al., “Align before Fuse: Vision and Language Representation Learning with Momentum Distillation,” Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021, 12 pages. [cited by applicant]
Li et al., “BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” Proceedings of the 39th International Conference on Machine Learning, 2022, 162:12888-12900. [cited by applicant]
Li et al., “Denoising Autoregressive Representation Learning,” CoRR, Mar. 8, 2024, arXiv:2403.05196, 22 pages. [cited by applicant]
Li et al., “Fine-Grained Semantically Aligned Vision-Language Pre-Training,” Advances in Neural Information Processing Systems 35 (NeurIPS 2022), Main Conference Track, 2022, 14 pages. [cited by applicant]
Li et al., “Oscar: Object-semantics aligned pre-training for vision language tasks,” Computer Vision—ECCV 2020: 16th European Conference, Glasgow, UK, Aug. 23-28, 2020, 21 pages. [cited by applicant]
Mizrahi et al., “4M: Massively Multimodal Masked Modeling,” CoRR, Dec. 11, 2023, arXiv:2312.06647, 46 pages. [cited by applicant]
OpenAI “GPT-4V(ision) System Card,” Computer Science, Engineering, Sep. 25, 2023, 18 pages. [cited by applicant]
Ordonez et al., “Im2Text: Describing Images Using 1 Million Captioned Photographs,” Advances in Neural Information Processing Systems 24 (NIPS 2011), 2011, 9 pages. [cited by applicant]
Parkhi et al., “Cats and Dogs,” 2012 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 16-21, 2012, pp. 3498-3505. [cited by applicant]
Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” Proceedings of the 38th International Conference on Machine Learning, 2021, 139:8748-8763. [cited by applicant]
Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” CoRR, Oct. 23, 2019, https://arxiv.org/abs/1910.10683, 67 pages. [cited by applicant]
Ramesh et al., “Zero-Shot Text-to-Image Generation,” Proceedings of the 38th International Conference on Machine Learning, 2021, 139:8821-8831. [cited by applicant]
Razavi et al., “Generating diverse high-fidelity images with VQ-VAE-2,” Advances in neural information processing systems (NeurIPS 2019), 2019, 11 pages. [cited by applicant]
Rombach et al., “High-Resolution Image Synthesis With Latent Diffusion Models,” CoRR, Dec. 20, 2021, arXiv:2112.10752, 45 pages. [cited by applicant]
Ronneberger et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation,” CoRR, May 18, 2015, arXiv:1505.04597, 8 pages. [cited by applicant]
Ruhu et al., “Rolling Diffusion Models,” CoRR, Feb. 12, 2024, arXiv:2402.09470, 18 pages. [cited by applicant]
Saharia et al., “Photorealistic text-to-image diffusion models with deep language understanding,” Advances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022, 16 pages. [cited by applicant]
Schuhmann et al., “LAION-400M: Open dataset of clip-filtered 400 million image-text pairs,” CoRR, Nov. 3, 2021, arXiv:2111.02114, 5 pages. [cited by applicant]
Shao et al., “Objects365: A large-scale, high-quality dataset for object detection,” IEEE/CVF international conference on computer vision, 2019, pp. 8430-8439. [cited by applicant]
Sharma, et al., “Conceptual Captions: A Cleaned, Hypernymed, Image A It-text Dataset for Automatic Image Captioning,” Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (vol. 1: Long… [cited by applicant]
Sohl-Dickstein et al., “Deep Unsupervised Learning using Nonequilibrium Thermodynamics,” Proceedings of the 32nd International Conference on Machine Learning, Jul. 7-9, 2015, 37:2256-2265. [cited by applicant]
Tian et al., “MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer,” CoRR, Jan. 18, 2024, arXiv:2401.10208, 35 pages. [cited by applicant]
Tong et al., “Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs,” CoRR, Jan. 11, 2024, arxiv.org/abs/2401.06209, 18 pages. [cited by applicant]
Touvron et al., “Llama 2: Open foundation and fine-tuned chat models,” CoRR, Jul. 18, 2023, arXiv:2307.09288, 77 pages. [cited by applicant]
Tschannen et al., “GIVT: Generative Infinite-Vocabulary Transformers,” CoRR, Dec. 4, 2023, arXiv:2312.02116, 32 pages. [cited by applicant]
Tschannen et al., “Image Captioners Are Scalable Vision Learners Too,” Thirty-seventh Conference on Neural Information Processing Systems, Sep. 21, 2023, 26 pages. [cited by applicant]
Van den Oord et al., “Neural discrete representation learning,” Advances in Neural Information Processing Systems 30, 2017, 10 pages. [cited by applicant]
Vaswani et al., “Attention Is All You Need,” CoRR, Jun. 12, 2017, arXiv:1706.0372, 15 pages. [cited by applicant]
Wang et al., “GIT: A Generative Image-to-text Transformer for Vision and Language,” Transactions on Machine Learning Research, Dec. 12, 2022, 18 pages. [cited by applicant]
Wang et al., “Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences,” CoRR, Jan. 19, 2024, arXiv:2401.10529, 27 pages. [cited by applicant]
Wang et al., “On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment,” Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Nov. 2020, pp. 4438… [cited by applicant]
Yu et al., “CoCa: Contrastive Captioners are Image-Text Foundation Models,” Transactions on Machine Learning Research, Aug. 26, 2022, 20 pages. [cited by applicant]
Yu et al., “Scaling Autoregressive Models for Content-Rich Text-to-Image Generation,” Transactions on Machine Learning Research, Nov. 7, 2022, 53 pages. [cited by applicant]
Yu et al., “Scaling autoregressive multi-modal models: Pretraining and instruction tuning,” CoRR, Sep. 5, 2023, arXiv:2309.02591, 19 pages. [cited by applicant]
Yu et al., “Vector-quantized Image Modeling with Improved VQGAN,” International Conference on Learning Representations, Jan. 28, 2022, 17 pages. [cited by applicant]
Zhai et al., “LiT: Zero-Shot Transfer with Locked-image Text Tuning,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 18123-18133. [cited by applicant]
Zhai et al., “Sigmoid Loss for Language Image Pre-Training,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 11975-11986. [cited by applicant]
Zhan et al., “Multimodal image synthesis and editing: A survey and taxonomy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Nov. 3, 2023, 45(12):15098-15119. [cited by applicant]
Zhao et al., “Blessing of Class Diversity in Pre-training,” Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, 2023, 206:283-305. [cited by applicant]