Modality adaptive information retrieval
In some embodiments, a multimodal computing system receives a query and identifies, from source documents, text passages and images that are relevant to the query. The multimodal computing system accesses a multimodal question-answering model that includes a textual stream of language models and a visual stream of language models. Each of the textual stream and the visual stream contains a set of transformer-based models and each transformer-based model includes a cross-attention layer using data generated by both the textual stream and visual stream of language models as an input. The multimodal computing system identifies text relevant to the query by applying the textual stream to the text passages and computes, using the visual stream, relevance scores of the images to the query, respectively. The multimodal computing system further generates a response to the query by including the text and/or an image according to the relevance scores.
1. A computer-implemented method, comprising:
receiving and storing in a memory device, by a multimodal query subsystem, a text-based query;
converting a set of images into a first embedding representation and one or more source documents into a second embedding representation;
determining, based on a comparison of the first embedding representation and the second embedding representation, by the multimodal query subsystem in one or more source documents, a text passage and one or more images from the set of images that are relevant to the text-based query;
accessing, by the multimodal query subsystem, a multimodal question-answering model comprising (a) a first stream of language models comprising a first set of transformer-based models concatenated with each other and (b) a second stream of language models comprising a second set of transformer-based models concatenated with each other, wherein each transformer- based model comprises a respective cross-attention layer using data generated by both the first stream of language models and the second stream of language models as input;
generating, by the multimodal query subsystem, an indication of a portion of the text passage that is relevant to the text-based query by, at least, applying the first stream of language models to the text passage;
computing and storing in the memory device, by the multimodal query subsystem and with the second stream of language models, relevance scores of the text-based query for the one or more images, respectively, wherein the relevance scores are computed based on data received from the first stream of language models via cross-attention layers of the second stream of language models;
generating, by the multimodal query subsystem, a response to the text-based query comprising at least one of (a) the portion of the text passage or (b) an image among the one or more images from the set of images according to the respective relevance scores; and
transmitting the response for display on a display device.
2. The computer-implemented method of claim 1 , wherein determining the text passage and the one or more images that are relevant to the text- based query comprises:
determining the text passage from a text portion of the one or more source documents;
extracting multiple images from the one or more source documents;
determining a similarity between each of the multiple images and the text passage; and
identifying the one or more of images that are relevant to the text-based query as images among the multiple images having a similarity higher than a pre- determined threshold.
3. The computer-implemented method of claim 1 , wherein each of the first set of transformer-based models in the first stream of language models further comprises a self-attention layer using data generated by the first stream of language models as an input.
4. The computer-implemented method of claim 3 , wherein each of the first set of transformer-based models in the first stream of language models and the second set of transformer-based models in the second stream of language models further comprises a feedforward neural network.
5. A computer-implemented method for generating a multimodal question-answering model for providing a multimodal answer to a query, the method comprising:
a training data generation module configured for generating and storing in a memory device training data for the multimodal question-answering model, wherein the multimodal question-answering model comprises a first stream of language models configured for text content and a second stream of language models configured for image content, generating the training data comprising:
accessing a dataset comprising queries and text-based answers for the respective queries;
identifying, from the queries in the dataset, a query whose text-based answer is contained in a document including both textual content and image content;
converting the image content from the document into an embedding representation;
extracting the image content from the embedding representation;
determining a relevance score indicating a relevance of the image content to the query using one or more embedding representations of the image content, a caption of the image content, the text-based answer of the query, or a source passage containing the text-based answer in the document; and
generating an entry of the training data that includes the query, the source passage, the image content, the text-based answer, and the relevance score of the image content; and
a model training module configured for training the multimodal question-answering model using the training data.
6. The computer-implemented method of claim 5 , wherein determining the relevance score comprises:
calculating a proximity distance between the image content and the source passage using a number of tokens as a distance unit;
calculating a first term frequency-inverse document frequency (TF-IDF) score of the caption of the image content with the query;
calculating a second TF-IDF score of the caption of the image content with the text-based answer;
calculating a third TF-IDF score of the caption of the image content with the source passage; and
computing the relevance score of the image content by combining the proximity distance, the first TF-IDF score of the caption, the second TF-IDF score of the caption, and
the third TF-IDF score of the caption.
7. The computer-implemented method of claim 6 , wherein a loss function used in training the multimodal question-answering model comprises a loss term for the image content, the loss term for the image content comprising a weighted binary cross-entropy with a weight as the relevance score.
8. The computer-implemented method of claim 5 , wherein:
the first stream of language models is configured to accept a query and a text passage as input and output an indication of a portion of the text passage that is relevant to the query; and
the second stream of language models is configured to accept multiple images as input and output a relevance score to the query for each of the multiple images.
9. The computer-implemented method of claim 5 , wherein the second stream of language models comprising a plurality of transformer-based models concatenated with each other, each of the plurality of transformer-based models comprising a cross-attention layer that uses data generated by both the first stream of language models and the second stream of language models as an input.
10. The computer-implemented method of claim 5 , further comprising a pre- training module configured for:
pre-training the first stream of language models using a first training dataset comprising queries and corresponding text-based answers; and
pre-training the second stream of language models using a second training dataset, wherein training the multimodal question-answering model by the model training module comprises training the pre-trained first stream of language models and the pre-trained second stream of language models.
11. The computer-implemented method of claim 10 , wherein the pre-training module is further configured for generating the second training dataset by:
accessing an image and a caption of the image;
associating an irrelevant image with the caption of the image; and
generating an entry of the second training dataset, the entry comprising the caption of the image, the image, an indication that the image is relevant to the caption, the irrelevant image, and an additional indication indicating the caption is irrelevant to the irrelevant image.
12. The computer-implemented method of claim 5 , wherein the first stream of language models comprises a plurality of transformer-based models concatenated with each other, each of the plurality of transformer-based models comprising (i) a self-attention layer using data generated by the first stream of language models as input and (ii) a cross-attention layer using data generated by both the first stream of language models and the second stream of language models as inputs.
13. The computer-implemented method of claim 12 , wherein each of the plurality of transformer-based models in the first stream of language models and a second plurality of transformer-based models in the second stream of language models comprises a feedforward neural network.
14. The computer-implemented method of claim 5 , wherein training the multimodal question-answering model comprises adjusting parameters of the multimodal question-answering model to minimize a loss function, wherein the loss function comprises a first term calculated for the first stream of language models and a second loss term calculated for the second stream of language models.
15. A system, comprising:
one or more processing devices; and
a non-transitory computer-readable medium having program code that is stored thereon, the program code executable by one or more processing devices for performing operations comprising:
identifying, from one or more source documents, a text passage and a plurality of images that are relevant to a query;
converting the text passage into a first embedding representation and the plurality of images into a second embedding representation;
applying, to the first embedding representation, a first stream of language models from a multimodal question-answering model to obtain an indication of a portion of the text passage that is relevant to the query;
applying, to the second embedding representation, a second stream of language models from the multimodal question-answering model to obtain relevance scores of the query for the plurality of images, respectively, wherein each of the first stream of language models and the second stream of language models comprises a set of transformer-based models concatenated with each other, each of the set of transformer-based models comprising a cross-attention layer using data generated by both the first stream of language models and the second stream of language models as input; and
generating a response to the query comprising at least one of (a) the portion of the text passage or (b) an image among the plurality of images according to the respective relevance scores; and
transmitting the response for display on a display device.
16. The system of claim 15 , wherein determining the text passage and the plurality of images that are relevant to the query comprises:
determining the text passage from a text portion of the one or more source documents;
extracting multiple images from the one or more source documents;
determining a similarity between each of the multiple images and the text passage; and
identifying the plurality of images that are relevant to the query as images among the multiple images having a similarity higher than a pre-determined threshold.
17. The system of claim 16 , wherein determining a similarity between each of the multiple images and the text passage comprises:
generating an embedding for each of the multiple images;
generating an embedding for the text passage; and
calculating the similarity as a similarity between the embedding of each of the multiple images and the embedding for the text passage.
18. The system of claim 15 , wherein each of the transformer-based models in the first stream of language models further comprises a self-attention layer using data generated by the first stream of language models as an input.
19. The system of claim 15 , wherein each of the transformer-based models in the first stream of language models and the second stream of language models further comprises a feedforward neural network.