IP Library › Granted Patent US 12700225
Granted Patent B2
US 12700225 · App. 18/476,037 · Granted Aug 4, 2026

Channel fusion for vision-language representation learning

Inventors: Anthony J. Piergiovanni (Denver, CO); Maxwell Mbabilla Aladago (Hanover, NH)
Assignee: GOOGLE LLC
G06V10/80G06V10/774G06V30/19147G06V30/1918
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12700225
App. No.
18/476,037
Granted
Aug 4, 2026
Kind
B2
Abstract

Provided is an approach that aligns multi-modal tokens using cross-attention without losing the advantages of global self-attention. In contrast to previous works that concatenate the unimodal tokens along the sequence dimension, example approaches described herein align per-modality tokens by chaining them along the channels. Specifically, the tokens from one modality can be used to query the other modality and the output can be concatenated with the query tokens on the channels. An analogous process can also be repeated (or performed in parallel) where the roles of the two modalities are switched. The resulting sets of compound tokens can be concatenated and fed into a self-attention encoder such as a transformer encoder that performs self-attention.

Claims (54)

1 . A computer system for performance of visuo-linguistic processing, the computer system comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store a machine-learned visuo-linguistic model, the machine-learned visuo-linguistic model comprising one or more multi-modal fusion layers, each multi-modal fusion layer configured to:

obtain a plurality of vision embedding tokens and a plurality of text embedding tokens;

perform cross-attention using the plurality of vision embedding tokens as queries and the plurality of text embedding tokens as keys and values to generate a plurality of text-to-vision intermediate tokens;

concatenate the plurality of text-to-vision intermediate tokens with the plurality of vision embedding tokens in a channel-wise fashion to generate a plurality of text-to-vision compound tokens;

perform cross-attention using the plurality of text embedding tokens as queries and the plurality of vision embedding tokens as keys and values to generate a plurality of vision-to-text intermediate tokens; and

concatenate the plurality of vision-to-text intermediate tokens with the plurality of text embedding tokens in a channel-wise fashion to generate a plurality of vision-to-text compound tokens.

2 . The computer system of claim 1 , wherein the machine-learned visuo-linguistic model is configured to concatenate the plurality of text-to-vision compound tokens and the plurality of vision-to-text compound tokens to form a first set of output compound tokens.

3 . The computer system of claim 2 , wherein the machine-learned visuo-linguistic model further comprises one or more self-attention layers configured to perform self-attention on the first set of output compound tokens to generate a second set of output compound tokens.

4 . The computer system of claim 3 , wherein the machine-learned visuo-linguistic model further comprises a decoder configured to process the second set of output compound tokens to generate a prediction for a visuo-linguistic task.

5 . The computer system of claim 4 , wherein the visuo-linguistic task comprises: a visual question answering task, a visual reasoning task, or a visual entailment task.

6 . The computer system of claim 2 , wherein the machine-learned visuo-linguistic model further comprises a decoder configured to process the first set of output compound tokens generate a prediction for a visuo-linguistic task.

7 . A computer system for multi-modal processing, the computer system comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store a machine-learned multi-modal model, the machine-learned multi-modal model comprising one or more multi-modal fusion layers, each multi-modal fusion layer configured to:

obtain a plurality of first embedding tokens associated with a first data modality;

obtain a plurality of second embedding tokens associated with a second data modality that is different from the first data modality;

perform cross-attention using the plurality of first embedding tokens as queries and the plurality of second embedding tokens as keys and values to generate a plurality of second-to-first intermediate tokens;

concatenate the plurality of second-to-first intermediate tokens with the plurality of first embedding tokens in a channel-wise fashion to generate a plurality of second-to-first compound tokens;

perform cross-attention using the plurality of second embedding tokens as queries and the plurality of first embedding tokens as keys and values to generate a plurality of first-to-second intermediate tokens; and

concatenate the plurality of first-to-second intermediate tokens with the plurality of second embedding tokens in a channel-wise fashion to generate a plurality of first-to-second compound tokens.

8 . The computer system of claim 7 , wherein the machine-learned multi-modal model is configured to concatenate the plurality of second-to-first compound tokens and the plurality of first-to-second compound tokens to form a first set of output compound tokens.

9 . The computer system of claim 8 , wherein the machine-learned multi-modal model further comprises one or more self-attention layers configured to perform self-attention on the first set of output compound tokens to generate a second set of output compound tokens.

10 . The computer system of claim 9 , wherein the machine-learned multi-modal model further comprises a decoder configured to process the second set of output compound tokens generate a prediction for a multi-modal task.

11 . The computer system of claim 8 , wherein the machine-learned multi-modal model further comprises a decoder configured to process the first set of output compound tokens generate a prediction for a multi-modal task.

12 . The computer system of claim 7 , wherein the first modality comprises visual data and the second modality comprises textual data.

13 . The computer system of any of claim 7 , wherein the first modality comprises visual data and the second modality comprises audio data.

14 . The computer system of any of claim 7 , wherein the first modality comprises audio data and the second modality comprises textual data.

15 . A computer system for multi-modal processing, the computer system comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store a machine-learned multi-modal model, the machine-learned multi-modal model comprising one or more multi-modal fusion layers, each multi-modal fusion layer configured to:

obtain a plurality of first embedding tokens associated with a first data modality;

obtain a plurality of second embedding tokens associated with a second data modality that is different from the first data modality;

perform cross-attention using the plurality of first embedding tokens as queries and the plurality of second embedding tokens as keys and values to generate a plurality of second-to-first intermediate tokens; and

concatenate the plurality of second-to-first intermediate tokens with the plurality of first embedding tokens in a channel-wise fashion to generate a plurality of second-to-first compound tokens;

wherein the machine-learned multi-modal model further comprises a decoder configured to process the plurality of second-to-first compound tokens to generate a prediction for a multi-modal task.

16 . The computer system of any of claim 15 , wherein:

the first modality comprises visual data and the second modality comprises textual data; or

the first modality comprises textual data and the second modality comprises visual data.

17 . The computer system of any of claim 15 , wherein:

the first modality comprises visual data and the second modality comprises audio data;

the first modality comprises audio data and the second modality comprises visual data;

the first modality comprises audio data and the second modality comprises textual data; or

the first modality comprises textual data and the second modality comprises audio data.

18 . A computer system for multi-modal processing, the computer system comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store a machine-learned multi-modal model, the machine-learned multi-modal model comprising one or more multi-modal fusion layers, each multi-modal fusion layer configured to:

obtain a plurality of first embedding tokens associated with a first data modality;

obtain a plurality of second embedding tokens associated with a second data modality that is different from the first data modality;

perform cross-attention using the plurality of first embedding tokens as queries and the plurality of second embedding tokens as keys and values to generate a plurality of second-to-first intermediate tokens; and

concatenate the plurality of second-to-first intermediate tokens with the plurality of first embedding tokens in a channel-wise fashion to generate a plurality of second-to-first compound tokens;

wherein the machine-learned multi-modal model further comprises one or more self-attention layers configured to perform self-attention on the plurality of second-to-first compound tokens to generate a set of output compound tokens; and

wherein the machine-learned multi-modal model further comprises a decoder configured to process the set of output compound tokens to generate a prediction for a multi-modal task.