Apparatus and method for diagnosing a medical condition from a medical image
An apparatus for diagnosing a medical condition including at least a processor and a memory, the memory containing instructions configuring the at least a processor to receive a medical image that is the result of a medical imaging procedure and train a medical image classifier and a medical report classifier. Training the classifiers includes receiving training data, including a plurality of prior medical images and a plurality of prior medical reports, training the medical image classifier and the medical report classifier, and optimizing both classifiers using a common loss function. The memory further containing instructions configuring the at least a processor to generate a label for the medical image, including inputting the medical image into the medical image classifier and receiving the label as output from the medical image classifier.
1. A method for diagnosing a medical condition, the method comprising:
receiving a medical image selected from the group consisting of radiographs, magnetic resonance imaging, nuclear medicine imaging, or ultrasounds, wherein the medical image is the result of a medical imaging procedure;
training a medical image encoder, a medical report encoder, and a classifier, wherein training the encoders and classifier comprises:
receiving training data, wherein the training data comprises a plurality of prior medical images and a plurality of prior medical reports, wherein each prior medical image of the plurality of prior medical images is associated with a prior medical report of the plurality of prior medical reports to form a pair of image-text embeddings, which extract bidirectional encoder representations using neural networks;
training the medical image encoder and the medical report encoder using the training data; and
optimizing the encoders using a ranking-based loss function of output embeddings of the medical image encoder and the medical report encoder; and
generating a label for the medical image, wherein generating the label for the medical image comprises:
inputting the medical image into the medical image encoder; and
receiving the label as output from the medical image encoder and the classifier.
2. The method of claim 1 , further comprising encoding the medical image, wherein encoding the medical image comprises extracting image features.
3. The method of claim 1 , further comprising encoding the prior medical report, wherein encoding the prior medical report comprises extracting text features.
4. The method of claim 3 , further comprising tokenizing the prior medical report, wherein tokenizing the prior medical report comprises splitting the medical report into words or letters and converting the words or letters to tokens.
5. The method of claim 1 , further comprising, decoupling the medical image encoder from the medical report encoder after training the medical image encoder and the medical report encoder.
6. The method of claim 1 , wherein the label comprises a severity of a medical condition.
7. The method of claim 1 , wherein the training data further comprises a plurality of associated labels, wherein each associated label is associated with an image-text pair.
8. The method of claim 1 , wherein:
the medical image is a chest radiograph; and
the prior medical report is a radiology report.
9. The method of claim 8 , wherein the label for the medical image is a severity label, wherein the severity label denotes the severity of a pulmonary edema.
10. The method of claim 1 , further comprising providing a textual explanation of the label received from the medical image classifier.
11. An apparatus for diagnosing a medical condition, the apparatus comprising:
at least a processor; and
a memory communicatively connected to the processor, the memory containing instructions configuring the at least a processor to:
receive a medical image selected from the group consisting of radiographs, magnetic resonance imaging, nuclear medicine imaging, or ultrasounds, and is, wherein the medical image is the result of a medical imaging procedure;
train a medical image encoder and a medical report encoder, wherein training the medical image encoder and the medical report encoder comprises:
receiving training data, wherein the training data comprises a plurality of prior medical images and a plurality of prior medical reports, wherein each prior medical image of the plurality of prior medical images is associated with a prior medical report of the plurality of prior medical reports to form a pair of image-text embeddings, which extract bidirectional encoder representations using neural networks;
training the medical image encoder and the medical report encoder using the training data; and
optimizing the encoders using a ranking-based loss function of output embeddings of the medical image encoder and the medical report encoder; and
generate a label for the medical image, wherein generating the label for the medical image comprises:
inputting the medical image into the medical image classifier; and
receiving the label as output from the medical image classifier.
12. The apparatus of claim 11 , wherein the memory contains instructions further configuring the processor to encode the medical image, wherein encoding the medical image comprises extracting image features.
13. The apparatus of claim 11 , wherein the memory contains instructions further configuring the processor to encode the prior medical report, wherein encoding the prior medical report comprises extracting text features.
14. The apparatus of claim 13 , wherein the memory contains instructions further configuring the processor to tokenize the prior medical report, wherein tokenizing the prior medical report comprises splitting the medical report into words or letters and converting the words or letters to tokens.
15. The apparatus of claim 11 , wherein the memory contains instructions further configuring the processor to decouple the medical image encoder from the medical report encoder after training the medical image encoder and the medical report encoder.
16. The apparatus of claim 11 , wherein the label comprises a severity of a medical condition.
17. The apparatus of claim 11 , wherein the training data further comprises a plurality of associated labels, wherein each associated label is associated with an image-text pair.
18. The apparatus of claim 11 , wherein:
the medical image is a chest radiograph; and
the prior medical report is a radiology report.
19. The apparatus of claim 18 , wherein the label for the medical image is a severity label, wherein the severity label denotes the severity of a pulmonary edema.
20. The apparatus of claim 11 , wherein the memory contains instructions further configuring the processor to provide a textual explanation of the label received from the medical image classifier.