Training multi-modal models on documents using multiple instance learning
An example system includes a processor to automatically extract text and images from a document. The processor can automatically generate text bags including a number of nearest texts for each of the extracted images. The processor can then train a multi-modal model based on the automatically generated text bags using a CLIP-MIL loss that computes, for each of the extracted images, a correlation between each of the different texts in the texts bags using a CLIP feature space at each gradient step of the gradient descent-based multiple instance learning (MIL) algorithm.
1 . A computer system, comprising:
a processor set;
one or more computer-readable storage media; and
program instructions stored on the one or more computer-readable storage media to cause the processor set to perform operations comprising:
automatically extracting text and images from a document;
automatically generating positive text bags comprising a plurality of collocated texts for each of the extracted images in the document;
automatically generating negative text bags comprising a plurality of collocated texts for extracted images in other documents; and
training a multi-modal model based on the automatically generated positive text bags using a Contrastive Language-Image Pre-Training (CLIP) multiple instance learning (MIL) loss that computes, for each of the extracted images, a correlation between each of the texts in the positive text bags using a CLIP feature space at each gradient step of a gradient descent-based MIL algorithm, wherein the CLIP-MIL loss uses negative text bags and a MIL-Softmax loss function that comprises an average weighted by similarity over each positive text bag.
2 . The computer system of claim 1 , wherein the CLIP-MIL loss comprises a MIL-MAX loss function that selects a text with maximal similarity from each positive text bag associated with an image.
3 . The computer system of claim 1 , wherein the average is normalized using both positive text matches and negative text matches.
4 . The computer system of claim 1 , wherein the CLIP-MIL loss comprises MIL-NCE loss with a function is adapted to a CLIP contrastive loss using the image text matches and text images positive match bags and negative match bags.
5 . The computer system of claim 1 , wherein the operations further comprise executing a low-rank adaptation of large models (LoRA) to fine-tune the multi-modal model by reducing a number of trainable parameters during training.
6 . The computer system of claim 5 , wherein the LoRA is adapted to work on a residual neural network (ResNet) of the multi-modal model, wherein each of a linear layer, a convolutional layer, and an embedding layer of the ResNet is adapted into a LoRA layer by a addition of a set of trainable parameters.
7 . The computer system of claim 5 , wherein the LoRA is applied to all layers of the multi-modal model.
8 . The computer system of claim 1 , wherein the processor is to train the multi-modal model using image-texts bags comprising an image and a plurality of associated proximate texts.
9 . The computer system of claim 1 , wherein the operations further comprise training the multi-modal model using text-images bags comprising a text and a plurality of associated images.
10 . A computer-implemented method, comprising: automatically generating a plurality of positive image-texts bags including number of collocated extracted texts for each extracted image from a document;
automatically generating negative text bags comprising a plurality of collocated texts for extracted images in other documents; and
fine-tuning an unlocked subset of parameters of a pretrained multi-modal model based on generated positive text bags using a Contrastive Language-Image Pre-Training (CLIP) multiple instance learning (MIL) loss that computes, for each extracted image, a correlation between different texts in positive text bags using a CLIP feature space at each gradient step of gradient descent-based MIL algorithm, wherein pretrained parameters of the pretrained multi-modal model are locked during the fine-tuning, wherein the CLIP-MIL loss uses negative text bags and a MIL-Softmax loss function that comprises an average weighted by similarity over each positive text bag.
11 . The computer-implemented method of claim 10 , further comprising pretraining the multi-modal model on a large set of image-text pairs.
12 . The computer-implemented method of claim 10 , wherein fine-tuning the unlocked subset of parameters comprises using a CLIP-MIL Max loss that selects a text with a highest correlation for each extracted image from the document.
13 . The computer-implemented method of claim 10 , wherein fine-tuning the unlocked subset of parameters comprises using a CLIP-MIL NCE loss that calculates a weighted average between texts in each of a plurality of positive text bags.
14 . The computer-implemented method of claim 13 , comprising normalizing the weighted average using both positive text matches and negative text matches.
15 . The computer-implemented method of claim 10 , wherein fine-tuning the unlocked subset of parameters comprises executing a low-rank adaptation of large models (LoRA) to reduce a number of trainable parameters during fine-tuning.
16 . The computer-implemented method of claim 10 , comprising receiving a text as input at the multi-modal model and retrieving an associated image via the multi-modal model.
17 . The computer-implemented method of claim 10 , comprising receiving an image as input at the multi-modal model and retrieving an associated text via the multi-modal model.
18 . A computer program product comprising:
one or more computer-readable storage media; and
program instructions stored on the one or more computer-readable storage media to perform operations comprising:
automatically generate a plurality of positive image-texts bags including number of collocated extracted texts for each extracted image from a document;
automatically generating negative text bags comprising a plurality of collocated texts for extracted images in other documents; and
fine-tuning an unlocked subset of parameters of a pretrained multi-modal model based on generated positive text bags using a Contrastive Language-Image Pre-Training (CLIP) multiple instance learning (MIL) loss that computes, for each extracted image, a correlation between different texts in positive text bags using a CLIP feature space at each gradient step of gradient descent-based MIL algorithm, wherein pretrained parameters of the pretrained multi-modal model are locked during the fine-tuning, wherein the CLIP-MIL loss uses negative text bags and a MIL-Softmax loss function that comprises an average weighted by similarity over each positive text bag.
19 . The computer program product of claim 18 , wherein the operations further comprise pretraining the multi-modal model on a large set of image-text pairs.