System for aggregating, analyzing, and reporting medical information
In one embodiment, system for aggregating, analyzing, and reporting medical information includes a front end module for managing a user interface, a back end module for exchanging patient information with a clinic record system and obtaining one or more medical images therefrom, a machine learning/artificial intelligence (ML/AI) engine for analyzing said one or more medical images and generating analysis results, and a report generator for generating a report that includes the analysis results. The ML/AI engine can include an anatomical plane classifier such as a 20+2 classifier, and the anatomical structure classifier can apply sematic segmentation.
1. A system for aggregating, analyzing, and reporting medical information comprising:
a front end module, operating on a computing device, for managing a user interface;
a back end module, operating on the computing device, for exchanging patient information with a clinic record system and obtaining one or more medical images therefrom;
a machine learning/artificial intelligence (ML/AI) engine, operating on the computing device, for analyzing said one or more medical images and generating analysis results, wherein the ML/AI engine further comprises:
a plane classification model that identifies and classifies fetal-cross sectional planes from the one or more medical images;
an anatomical structural classifier that applies semantic segmentation to associate specific pixels of the one or more images with corresponding anatomical structures of a given fetal-cross-sectional plane by classifying each pixel; and
an abnormality flagging model that classifies the one or more images as abnormal and identifies a subset of the specific pixels associated with an anatomical structure as abnormal;
a report generator, operating on the computing device, for generating a report that includes said analysis results.
2. The system of claim 1 , wherein the machine learning/artificial intelligence (ML/AI) engine applies rules that are encoded as weight matrices configured to operate on fetal ultrasound images.
3. The system of claim 2 , wherein the weigh matrices are generated by:
a) pre-training on a large general public image dataset,
b) additional pre-training on a general fetal ultrasound image dataset using self-supervised learning,
c) re-architecting final layers of a pre-trained convolutional neural network from a) and b), and
d) creating multiple output heads to classify fetal planes, fetal abnormalities, fetal anatomical structures, and diagnostic utility, and
e) fine-tuning using an annotated dataset of second trimester fetal ultrasound images.
4. The system of claim 1 , wherein the plane classifier is an anatomical plane classifier and the given fetal-cross sectional plane is an anatomical plane.
5. The system of claim 4 , wherein the anatomical plane classifier is a 20+2 classifier.
6. The system of claim 1 , wherein the back end module includes API functionality activated from front end module.
7. The system of claim 1 , wherein the clinic record system is one or both of a PACS and EMR system.
8. The system of claim 1 , wherein the exchange of patient information is by way of a DICOMWeb API.
9. The system of claim 1 , wherein the patient information includes DICOM studies and/or DICOM structured reports.
10. The system of claim 1 , wherein the ML/AI provides a timestamp output of analyzed images.
11. The system of claim 1 , wherein the ML/AI engine is operable to flag images with quality issues.
12. The system of claim 1 , wherein the report pertains to second trimester anatomical ultrasound scans.
13. The system of claim 1 , wherein the report includes template-based information.
14. A method for generating an anatomical scan report comprising:
selecting a patient;
obtaining a fetal scan associated with the patient;
conducting an analysis of the fetal scan using a machine learning/artificial intelligence (ML/AI) engine operating on a computing device and having multiple output heads further comprising:
an anatomical plane classification model that identifies and classifies a fetal-cross-section plane from the fetal scan, wherein the anatomical plane classification model is a resnet model;
an anatomical structure classifier that associates specific pixels of the fetal scan with corresponding anatomical structures within the fetal scan by classifying each pixel, wherein the anatomical structure classifier is a Unet model; and
an abnormality flagging model that classifies the fetal scan as abnormal and identifies a subset of the specific pixels as abnormal in response to receiving the fetal scan and fetal-cross sectional plane, wherein the abnormality flagging model is another resnet model; and
generating a report that includes results of said analysis including at least classification of of the image as abnormal.
15. The method of claim 14 , wherein conducting an analysis of the fetal scan using the machine learning/artificial intelligence (ML/AI) engine comprises applying rules that are encoded as weight matrices configured to operate on fetal ultrasound images.
16. The method of claim 15 , wherein the weight matrices are generated by:
a) pre-training on a large general public image dataset,
b) additional pre-training on a general fetal ultrasound image dataset using self-supervised learning,
c) re-architecting final layers of a pre-trained convolutional neural network from a) and b), and
d) creating the multiple output heads to classify fetal planes, fetal abnormalities, fetal anatomical structures, and diagnostic utility, and
e) fine-tuning using an annotated dataset of second trimester fetal ultrasound images.
17. The method of claim 14 , wherein said obtaining is from a clinic record system selected from one or both of a PACS and EMR system.
18. The method of claim 14 , wherein the ML/AI engine provides a timestamp output of analyzed images.
19. The method of claim 14 , wherein the ML/AI engine is operable to flag images with quality issues.
20. A method for generating an anatomical scan report comprising:
selecting a patient;
obtaining a fetal scan associated with the patient;
conducting an analysis of the fetal scan using a machine learning/artificial intelligence (ML/AI) engine operating on a computing device and having multiple output heads further comprising:
an anatomical plane classification model that identifies and classifies a fetal-cross-section plane from the fetal scan, wherein the anatomical plane classification model is a resnet model having a first output head;
an anatomical structure classifier that associates specific pixels of the fetal scan with corresponding anatomical structures within the fetal scan by classifying each pixel of the fetal scan, wherein the anatomical structure classifier is a Unet model having a second output head; and
an abnormality flagging model that outputs a classification of the fetal scan as abnormal and a heatmap of the fetal scan marking pixels of the fetal scan as relevant to the abnormality classification in response receiving the fetal scan and fetal-cross sectional plane, wherein the abnormality flagging model is another resnet model having a third output head; and
generating a report that includes results of said analysis including at least classification of of the image as abnormal.