Combined vision and language learning models for automated medical reports generation
A method of generating a medical report is presented herein. In some embodiments, the method includes receiving a medical image and at least one natural language medical question, extracting at least one image feature from the image; extracting at least one text feature from the question; and fusing the at least one image feature with the at least one text feature to form a combined feature. Some embodiments further include encoding, by an encoder, the combined feature to form a transformed combined feature; computing a set of prior context features based on a similarity between the transformed combined feature and each of a set of transformed text features derived from a set of training natural language answers; and generating, by a decoder, a first natural language answer conditioned on the transformed combined feature and the set of prior context features.
1 . A method of generating a medical report, the method comprising:
receiving a medical image and at least one natural language medical question;
extracting at least one image feature from the medical image;
extracting at least one text feature from the natural language medical question;
fusing the at least one image feature with the at least one text feature to form a combined feature;
encoding, by an encoder, the combined feature to form a transformed combined feature;
computing a set of prior context features based on a similarity between the transformed combined feature and each of a set of transformed text features derived from a set of training natural language answers; and
generating, by a decoder, a first natural language answer conditioned on the transformed combined feature and the set of prior context features.
2 . The method of claim 1 , wherein the extracting the at least one text feature is based on a pre-trained natural language model configured to predict masked words locally based on a context provided by other non-masked words in stored natural language medical questions and corresponding natural language answers.
3 . The method of claim 1 , wherein the encoder includes a first encoding projection matrix used to generate the transformed combined feature.
4 . The method of claim 1 , wherein computing the set of prior context features includes:
receiving a set of training natural language answers, each corresponding to one of a set of training medical images and one of a set of training natural language questions;
extracting a set of text features from the set of training natural language answers;
applying a second encoding projection matrix to the set of text features to generate the set of transformed text features;
storing the set of transformed text features in a database; and
maximizing a cosine similarity between the transformed combined feature and the set of transformed text features in the database using a top-k nearest context neighbor search.
5 . The method of claim 1 , wherein encoding the combined feature includes:
augmenting text data of the natural language medical question using a transformation function;
filtering punctuations from the augmented text data;
tokenizing each word in the augmented text data to generate tokenized text data; and
extracting the at least one text feature from the tokenized text data.
6 . The method of claim 1 , wherein encoding the combined feature is implemented at least in part by a pre-trained encoder that includes a plurality of transformer-based encoder blocks, and wherein each encoder block includes at least a bidirectional self-attention layer and a feed-forward layer.
7 . The method of claim 1 , wherein fusing the at least one image feature with the at least one text feature is implemented at least in part by a bilinear attention network model.
8 . The method of claim 1 , wherein predicting the set of natural language answers is implemented at least in part by a pre-trained decoder that includes a plurality of transformer-based decoder blocks, and wherein each decoder block includes at least a unidirectional self-attention layer and a feed-forward layer.
9 . The method of claim 1 , wherein the encoder has been trained using a set of training medical images, a set of natural language medical questions, and the set of training natural language answers each corresponding to one of the set of questions.
10 . A system for generating a medical report comprising:
an input interface configured to receive a medical image and at least one natural language medical question;
a memory configured to store a plurality of processor-executable instructions, the memory including:
an encoder module having an image encoder, a text encoder, a bilinear attention network, and an encoder built upon a contrastive learning model; and
a decoder module; and
a processor configured to execute the plurality of processor-executable instructions to perform operations including:
encoding, by the image encoder, the medical image to form a set of image features;
encoding, by the text encoder, the natural language medical question to form a set of first text features;
fusing, by the bilinear attention network, each image feature in the set of image features with each first text feature in the set of first text features to form a set of combined features;
encoding, by the encoder built on the contrastive learning model, each combined feature in the set of combined features to form a corresponding transformed combined feature;
computing a set of prior context features based on a similarity between the transformed combined feature and each of a set of transformed text features derived from a set of training natural language answers; and
predicting, by the decoder module, a first natural language answer conditioned on the transformed combined feature and the set of prior context features.
11 . The system of claim 10 , wherein computing the set of prior context features comprises:
receiving a set of training natural language answers, each corresponding to one of a set of training medical images and one of a set of training natural language questions;
extracting a set of text features from the set of training natural language answers;
applying an encoding projection matrix to the set of text features to generate the set of transformed text features;
storing the set of transformed text features in a database; and
maximizing a cosine similarity between the transformed combined feature and the set of transformed text features in the database using a top-k nearest context neighbor search.
12 . The system of claim 10 , wherein encoding each combined feature includes:
augmenting text data of the natural language medical question using a transformation function;
filtering punctuations from the augmented text data;
tokenizing each word in the augmented text data to generate tokenized text data; and
extracting the set of first text features from the tokenized text data.
13 . The system of claim 10 , wherein encoding each combined feature in the set of combined features is implemented at least in part by a pre-trained encoder that includes a plurality of transformer-based encoder blocks, and wherein each encoder block includes at least a bidirectional self-attention layer and a feed-forward layer.
14 . The system of claim 13 , wherein the bidirectional self-attention layer is configured to connect a text feature in the natural language medical question with all other text features of the natural language medical question to form context-dependent text features.
15 . The system of claim 10 , wherein the encoder has been trained using a set of training medical images, a set of natural language medical questions, and the set of training natural language answers each corresponding to one of the set of questions.
16 . A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for responding to questions about medical images, the plurality of processor-executable instructions being executed by a processor to perform operations comprising:
receiving a medical image and at least one natural language medical question;
extracting a set of image features from the medical image;
extracting at least one text feature from the at least one natural language medical question;
fusing the image features and the at least one text feature to form a set of combined features;
encoding, by an encoder, the set of combined features to form a set of transformed combined features;
computing a set of prior context features based on a similarity between the set of transformed combined features and a set of transformed text features derived from a set of training natural language answers; and
predicting, by a decoder, a set of descriptive natural language answers conditioned on the set of transformed combined features and the set of prior context features.
17 . The storage medium of claim 16 , wherein computing the set of prior context features includes:
storing the set of transformed text features in a database; and
maximizing a cosine similarity between the set of transformed combined features and the set of transformed text features in the database using a top-k nearest context neighbor search.
18 . The storage medium of claim 16 , further comprising:
projecting, by the decoder, the set of transformed combined features to a hidden space; and
mapping the projected set of transformed combined features to a set of classification categories corresponding to a subset of the questions.
19 . The storage medium of claim 16 , further comprising:
receiving the set of training natural language answers;
extracting a second set of text features from the set of training natural language answers; and
encoding, by the encoder, the second set of text features to generate the set of transformed text features.
20 . The storage medium of claim 16 , wherein the encoder includes a first encoding projection matrix used to generate the set of transformed combined features.