IP Library › Granted Patent US 12664708
Granted Patent B1
US 12664708 · App. 19/245,912 · Granted Jun 23, 2026

Multi-modal response generation

Inventors: Prafulla Dhariwal (San Francisco, CA); Gabriel Goh (San Francisco, CA); Ishaan Gulrajani (San Francisco, CA)
Assignee: OpenAI OpCo, LLC.
G06T11/60
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12664708
App. No.
19/245,912
Granted
Jun 23, 2026
Kind
B1
Abstract

The present technology pertains to systems and methods for improving efficiency in multi-modal AI models, particularly when handling long token sequences. Aspects of the present technology facilitate attention across large token sets (e.g., making up an image) by using border tokens output by an AI model at the beginning of respective rows of a set of image tokens. Thus, the AI model can reference an image token's position based on its position from a border token. Further, aspects of the present technology facilitate training of multi-model AI models to reduce negative influence between modalities.

Claims (61)

1 . A method comprising:

receiving, by an artificial intelligence (AI) model, a prompt to generate a multi-modal response, wherein the AI model is a multi-modal transformer model;

calculating query, key, and value (Q/K/V) vectors for tokens of a first modality using a first modality-specific weighting;

calculating Q/K/V vectors for tokens of a second modality using a second modality-specific weighting, wherein the first modality-specific weighting and the second modality-specific weighting are decoupled;

performing an attention operation across the Q/K/V vectors for the tokens of the first modality and the Q/K/V vectors for the tokens of the second modality to result in an attended embedding, wherein the attended embedding is based on the first modality-specific weighting and the second modality-specific weighting;

generating, by the AI model, a stream of tokens for the first modality using the attended embedding; and

outputting, by the AI model, the stream of tokens of the first modality to the user account.

2 . The method of claim 1 , further comprising:

generating, by the AI model, a stream of tokens of the second modality using the attended embedding; and

outputting, by the AI model, the stream of tokens of the second modality to the user account.

3 . The method of claim 1 , further comprising:

training the AI model by:

receiving, by the AI model, a training dataset comprising multi-modal interleaved data, and

processing, by the AI model, data of respective modalities to generate modality-specific weighting for the respective modalities.

4 . The method of claim 3 , wherein the multimodal interleaved data comprises coupled sets of tokens and wherein the multi-modal interleaved data is associated with a single context.

5 . The method of claim 3 , wherein calculating Q/K/V vectors for the respective modalities comprises:

passing, by the AI model, input tokens of the respective modalities through modality-specific projection matrices to create the Q/K/V vectors for the respective modalities, wherein the input tokens comprise the training dataset; and

performing, by the AI model, an attention operation over the input tokens to produce the attended embedding based on the Q/K/V vectors for the respective modalities.

6 . The method of claim 1 , wherein the tokens of the first modality are image tokens and wherein the image tokens comprise at least one border token.

7 . The method of claim 6 , wherein the at least one border token indicates a beginning of a row of an image and wherein the at least one border token is useable by an attention layer of the AI model to reference a particular image token in the row.

8 . A computing system comprising:

at least one processor; and

a memory storing instructions that, when executed by the at least one processor, configure the computing system to:

receive a prompt to generate a multi-modal response using an artificial intelligence (AI) model;

calculate query, key, and value (Q/K/V) vectors for tokens of a first modality using a first modality-specific weighting;

calculate Q/K/V vectors for tokens of a second modality using a second modality-specific weighting, wherein the first modality-specific weighting and the second modality-specific weighting are decoupled;

perform an attention operation across the Q/K/V vectors for the tokens of the first modality and the Q/K/V vectors for the tokens of the second modality to result in an attended embedding, wherein the attended embedding is based on the first modality-specific weighting and the second modality-specific weighting;

generate a stream of tokens for the first modality using the attended embedding;

output the stream of tokens of the first modality to the user account.

9 . The computing system of claim 8 , wherein the instructions further configure the computing system to:

generate, by the AI model, a stream of tokens of the second modality using the attended embedding; and

output the stream of tokens of the second modality to the user account.

10 . The computing system of claim 8 , wherein the instructions further configure the computing system to:

train the AI model for generating the multi-model response by:

receiving, by the AI model, a training dataset comprising multi-modal interleaved data, and

processing, by the AI model, data of respective modalities to generate modality-specific weighting for the respective modalities.

11 . The computing system of claim 10 , wherein the multimodal interleaved data comprises coupled sets of tokens and wherein the multi-modal interleaved data is associated with a single context.

12 . The computing system of claim 10 , wherein calculating Q/K/V vectors for the respective modalities comprises:

passing, by the AI model, input tokens of the respective modalities through modality-specific projection matrices to create the Q/K/V vectors for the respective modalities, wherein the input tokens comprise the training dataset; and

performing, by the AI model, an attention operation over the input tokens to produce the attended embedding based on the Q/K/V vectors for the respective modalities.

13 . The computing system of claim 8 , wherein the tokens of the first modality are image tokens and wherein the image tokens comprise at least one border token.

14 . The computing system of claim 13 , wherein the at least one border token indicates a beginning of a row of an image and wherein the at least one border token is useable by an attention layer of the AI model to reference a particular image token in the row.

15 . A non-transitory computer-readable medium comprising instructions that when executed by at least one processor, cause the at least one processor to:

receive a prompt to generate a multi-modal response using an artificial intelligence (AI) model;

calculate query, key, and value (Q/K/V) vectors for tokens of a first modality using a first modality-specific weighting;

calculate Q/K/V vectors for tokens of a second modality using a second modality-specific weighting, wherein the first modality-specific weighting and the second modality-specific weighting are decoupled;

perform an attention operation across the Q/K/V vectors for the tokens of the first modality and the Q/K/V vectors for the tokens of the second modality to result in an attended embedding, wherein the attended embedding is based on the first modality-specific weighting and the second modality-specific weighting;

generate a stream of tokens for the first modality using the attended embedding;

output the stream of tokens of the first modality to the user account.

16 . The non-transitory computer-readable medium of claim 15 , wherein the instructions further configure the at least one processor to:

generate, by the AI model, a stream of tokens of the second modality using the attended embedding; and

output the stream of tokens of the second modality to the user account.

17 . The non-transitory computer-readable medium of claim 15 , wherein the instructions further configure the at least one processor to:

train the AI model for generating the multi-model response by:

receiving, by the AI model, a training dataset comprising multi-modal interleaved data, and

processing, by the AI model, data of respective modalities to generate modality-specific weighting for the respective modalities.

18 . The non-transitory computer-readable medium of claim 17 , wherein the multimodal interleaved data comprises coupled sets of tokens and wherein the multi-modal interleaved data is associated with a single context.

19 . The non-transitory computer-readable medium of claim 17 , wherein calculating Q/K/V vectors for the respective modalities comprises:

passing, by the AI model, input tokens of the respective modalities through modality-specific projection matrices to create the Q/K/V vectors for the respective modalities, wherein the input tokens comprise the training dataset; and

performing, by the AI model, an attention operation over the input tokens to produce the attended embedding based on the Q/K/V vectors for the respective modalities.

20 . The non-transitory computer-readable medium of claim 15 , wherein the tokens of the first modality are image tokens and wherein the image tokens comprise at least one border token.