COMBINED VISION AND LANGUAGE LEARNING MODELS FOR AUTOMATED MEDICAL REPORTS GENERATION

1 . (canceled)

2 . A method of generating a medical report, the method comprising:

receiving image data of a medical image and natural language medical text;

extracting at least one image feature from the image data of the medical image;

extracting at least one text feature from the natural language medical text;

fusing the at least one image feature with the at least one text feature to form a combined feature;

encoding, by an encoder, the combined feature to form a transformed combined feature;

retrieving a set of prior context features from a set of transformed text features derived from a set of training natural language inferences; and

generating, by a decoder, a first natural language inference conditioned on the transformed combined feature and the set of prior context features.

3 . The method of claim 2 , wherein the extracting the at least one text feature is based on a pre-trained natural language model configured to predict masked words locally based on a context provided by other non-masked words in stored natural language medical texts and corresponding natural language inferences.

4 . The method of claim 2 , wherein the encoder includes a first encoding projection matrix used to generate the transformed combined feature.

5 . The method of claim 2 , wherein computing the set of prior context features includes:

receiving a set of training natural language inferences, each corresponding to one of a set of training medical images and one of a set of training natural language texts;

extracting a set of text features from the set of training natural language inferences;

applying a second encoding projection matrix to the set of text features to generate the set of transformed text features;

storing the set of transformed text features in a database; and

maximizing a cosine similarity between the transformed combined feature and the set of transformed text features in the database using a top-k nearest context neighbor search.

6 . The method of claim 2 , wherein encoding the combined feature includes:

augmenting text data of the natural language medical text using a transformation function;

filtering punctuations from the augmented text data;

tokenizing each word in the augmented text data to generate tokenized text data; and

extracting the at least one text feature from the tokenized text data.

7 . The method of claim 2 , wherein encoding the combined feature is implemented at least in part by a pre-trained encoder that includes a plurality of transformer-based encoder blocks, and wherein each encoder block includes at least a bidirectional self-attention layer and a feed-forward layer.

8 . The method of claim 2 , wherein fusing the at least one image feature with the at least one text feature is implemented at least in part by a bilinear attention network model.

9 . The method of claim 2 , wherein predicting the set of natural language inferences is implemented at least in part by a pre-trained decoder that includes a plurality of transformer-based decoder blocks, and wherein each decoder block includes at least a unidirectional self-attention layer and a feed-forward layer.

10 . The method of claim 2 , wherein the encoder has been trained using a set of training medical images, a set of natural language medical texts, and the set of training natural language inferences each corresponding to one of the set of texts.

11 . A system for generating a medical report comprising:

an input interface configured to receive image data of a medical image and natural language medical text;

a memory configured to store a plurality of processor-executable instructions, the memory including:

an encoder module having an image encoder, a text encoder, a bilinear attention network, and an encoder built upon a contrastive learning model; and

a decoder module; and

a processor configured to execute the plurality of processor-executable instructions to perform operations including:

encoding, by the image encoder, the image data of the medical image to form a set of image features;

encoding, by the text encoder, the natural language medical text to form a set of first text features;

fusing, by the bilinear attention network, each image feature in the set of image features with each first text feature in the set of first text features to form a set of combined features;

encoding, by the encoder built on the contrastive learning model, each combined feature in the set of combined features to form a corresponding transformed combined feature;

determining a set of prior context features from a set of transformed text features derived from a set of training natural language inferences; and

predicting, by the decoder module, a first natural language inference conditioned on the transformed combined feature and the set of prior context features.

12 . The system of claim 11 , wherein computing the set of prior context features comprises:

receiving a set of training natural language inferences, each corresponding to one of a set of training medical images and one of a set of training natural language texts;

extracting a set of text features from the set of training natural language inferences;

applying an encoding projection matrix to the set of text features to generate the set of transformed text features;

storing the set of transformed text features in a database; and

maximizing a cosine similarity between the transformed combined feature and the set of transformed text features in the database using a top-k nearest context neighbor search.

13 . The system of claim 11 , wherein encoding each combined feature includes:

augmenting text data of the natural language medical text using a transformation function;

filtering punctuations from the augmented text data;

tokenizing each word in the augmented text data to generate tokenized text data; and

extracting the set of first text features from the tokenized text data.

14 . The system of claim 11 , wherein encoding each combined feature in the set of combined features is implemented at least in part by a pre-trained encoder that includes a plurality of transformer-based encoder blocks, and wherein each encoder block includes at least a bidirectional self-attention layer and a feed-forward layer.

15 . The system of claim 14 , wherein the bidirectional self-attention layer is configured to connect a text feature in the natural language medical text with all other text features of the natural language medical text to form context-dependent text features.

16 . The system of claim 11 , wherein the encoder has been trained using a set of training medical images, a set of natural language medical texts, and the set of training natural language inferences each corresponding to one of the set of texts.

17 . A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for responding to texts about medical images, the plurality of processor-executable instructions being executed by a processor to perform operations comprising:

receiving image data of a medical image and natural language medical text;

extracting a set of image features from image data of the medical image;

extracting at least one text feature from the natural language medical text;

fusing the image features and the at least one text feature to form a set of combined features;

encoding, by an encoder, the set of combined features to form a set of transformed combined features;

determining a set of prior context features from a set of transformed text features derived from a set of training natural language inferences; and

predicting, by a decoder, a set of descriptive natural language inferences conditioned on the set of transformed combined features and the set of prior context features.

18 . The storage medium of claim 17 , wherein computing the set of prior context features includes:

storing the set of transformed text features in a database; and

maximizing a cosine similarity between the set of transformed combined features and the set of transformed text features in the database using a top-k nearest context neighbor search.

19 . The storage medium of claim 17 , further comprising:

projecting, by the decoder, the set of transformed combined features to a hidden space; and

mapping the projected set of transformed combined features to a set of classification categories corresponding to a subset of the texts.

20 . The storage medium of claim 17 , further comprising:

receiving the set of training natural language inferences;

extracting a second set of text features from the set of training natural language inferences; and

encoding, by the encoder, the second set of text features to generate the set of transformed text features.

21 . The storage medium of claim 17 , wherein the encoder includes a first encoding projection matrix used to generate the set of transformed combined features.