IP Library › Granted Patent US 12,236,592
Granted Patent B2
US 12,236,592 · App. 17/944,881 · Granted Feb 25, 2025

Systems, methods, and apparatuses for systematically determining an optimal approach for the computer-aided diagnosis of a pulmonary embolism

Inventors: Nahid Ul Islam (Mesa, AZ); Shiv Gehlot (Darwara, IN); Zongwei Zhou (Tempe, AZ); Jianming Liang (Scottsdale, AZ)
Assignee: Arizona Board of Regents on Behalf of Arizona State University
G06T7/0012A61B6/50G06T3/40G06V10/7715G06V10/774G06V10/82G16H50/20G06T2207/20081G06T2207/20084G06T2207/30061G06T2207/30101G06V2201/03
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,236,592
App. No.
17/944,881
Granted
Feb 25, 2025
Kind
B2
Abstract

Described herein are means for systematically determining an optimal approach for the computer-aided diagnosis of a pulmonary embolism, in the context of processing medical imaging. According to a particular embodiment, there is a system specially configured for diagnosing a Pulmonary Embolism (PE) within new medical images which form no part of the dataset upon which the AI model was trained. Such a system executes operations for receiving a plurality of medical images and processing the plurality of medical images by executing an image-level classification algorithm to determine the presence or absence of a Pulmonary Embolism (PE) within each image via operations including: pre-training an AI model through supervised learning to identify ground truth; fine-tuning the pre-trained AI model specifically for PE diagnosis to generate a pre-trained PE diagnosis and detection AI model; wherein the pre-trained AI model is based on a modified CNN architecture having introduced therein a squeeze and excitation (SE) block enabling the CNN architecture to extract informative features from the plurality of medical images by fusing spatial and channel-wise information; applying the pre-trained PE diagnosis and detection AI model to new medical images to render a prediction as to the presence or absence of the Pulmonary Embolism within the new medical images; and outputting the prediction as a PE diagnosis for a medical patient.

Claims (68)

1. A system comprising:

a memory to store instructions;

a processor to execute the instructions stored in the memory;

wherein the system is specially configured to:

receive a plurality of medical images;

process the plurality of medical images by executing an image-level classification algorithm to determine the presence or absence of Pulmonary Embolism (PE) within each image;

pre-train an AI model through supervised learning to identify ground truth;

fine-tune the pre-trained AI model specifically for PE diagnosis to generate a pre-trained PE diagnosis and detection AI model;

wherein the pre-trained AI model is based on a modified CNN architecture having introduced therein a squeeze and excitation (SE) block enabling the CNN architecture to extract informative features from the plurality of medical images by fusing spatial and channel-wise information;

apply the pre-trained PE diagnosis and detection AI model to new medical images to render a prediction as to the presence or absence of a Pulmonary Embolism within the new medical images; and

output the prediction as a PE diagnosis for a medical patient.

2. The system of claim 1 , wherein the new medical images constitute no part of any training set of medical images utilized to pre-train or fine-tune the AI model and have not been encountered by the pre-trained AI model prior to applying the pre-trained PE diagnosis and detection AI model to the new medical images.

3. The system of claim 1 , wherein the system is further specially configured to:

pre-train a vision transformer (ViT) architecture with a training image dataset; and

upscale each image in the training image dataset for a given patch size to increase the number of patches resulting in an enlarged size of the training image dataset.

4. The system of claim 3 , wherein the system is further specially configured to:

pre-train an AI model using different sized images generated from the upscaling of each image in the training image dataset for a given patch size.

5. The system of claim 1 , wherein applying the pre-trained PE diagnosis and detection AI model to new medical images to render a prediction as to the presence or absence of a Pulmonary Embolism within the new medical images comprises:

applying an image-level classification procedure individually to each one of the new medical images to determine, for each one of the new medical images, either the Pulmonary Embolism is present within the respective one of the new medical images or the Pulmonary Embolism is absent from the respective one of the new medical images.

6. The system of claim 1 , wherein pre-training an AI model through supervised learning to identify ground truth comprises applying pre-training via self-supervised learning using a publically available dataset having training images annotated as either PE presence or PE absence.

7. The system of claim 1 , wherein the system to receive a plurality of medical images comprises executing instructions for receiving a training dataset having included therein the plurality of medical images;

wherein the system is further specially configured to enlarge the training dataset's size beyond a received size for the training dataset by:

reshaping each of the plurality of images of the training dataset into a sequence of patches;

upscaling a corresponding image for each patch within the sequence of patches to generate an increased quantity of patches; and

wherein the system to pre-train the AI model through supervised learning to identify ground truth comprises the system executing instructions for pre-training the AI model through supervised learning using the increased quantity of patches as an enlarged training dataset.

8. A computer-implemented method performed by a system having at least a processor and a memory therein, wherein the computer-implemented method comprises:

executing instructions via the processor of the system for receiving a plurality of medical images at a receive interface of the system;

executing instructions via the processor of the system for processing the plurality of medical images by executing an image-level classification algorithm to determine the presence or absence of a Pulmonary Embolism (PE) within each image;

pre-training an AI model through supervised learning to identify ground truth;

fine-tuning the pre-trained AI model specifically for PE diagnosis to generate a pre-trained PE diagnosis and detection AI model;

wherein the pre-trained AI model is based on a modified CNN architecture having introduced therein a squeeze and excitation (SE) block enabling the CNN architecture to extract informative features from the plurality of medical images by fusing spatial and channel-wise information;

applying the pre-trained PE diagnosis and detection AI model to new medical images to render a prediction as to the presence or absence of the Pulmonary Embolism within the new medical images; and

outputting the prediction as a PE diagnosis for a medical patient.

9. The computer-implemented method of claim 8 , wherein the new medical images constitute no part of any training set of medical images utilized to pre-train or fine-tune the AI model and have not been encountered by the pre-trained AI model prior to applying the pre-trained PE diagnosis and detection AI model to the new medical images.

10. The computer-implemented method of claim 8 , further comprising:

pre-training a vision transformer (ViT) architecture with a training image dataset; and

upscaling each image in the training image dataset for a given patch size to increase the number of patches resulting in an enlarged size of the training image dataset.

11. The computer-implemented method of claim 10 , further comprising:

pre-training an AI model using different sized images generated from the upscaling of each image in the training image dataset for a given patch size.

12. The computer-implemented method of claim 8 , wherein applying the pre-trained PE diagnosis and detection AI model to new medical images to render a prediction as to the presence or absence of a Pulmonary Embolism within the new medical images comprises:

applying an image-level classification procedure individually to each one of the new medical images to determine, for each one of the new medical images, either the Pulmonary Embolism is present within the respective one of the new medical images or the Pulmonary Embolism is absent from the respective one of the new medical images.

13. The computer-implemented method of claim 8 , wherein pre-training an AI model through supervised learning to identify ground truth comprises applying pre-training via self-supervised learning using a publically available dataset having training images annotated as either PE presence or PE absence.

14. The computer-implemented method of claim 8 , wherein receiving a plurality of medical images comprises executing instructions for receiving a training dataset having included therein the plurality of medical images; and

wherein the computer-implemented method further enlarges the training dataset's size beyond a received size for the training dataset by:

reshaping each of the plurality of images of the training dataset into a sequence of patches;

upscaling a corresponding image for each patch within the sequence of patches to generate an increased quantity of patches; and

wherein pre-training the AI model through supervised learning to identify ground truth comprises executing instructions for pre-training the AI model through supervised learning using the increased quantity of patches as an enlarged training dataset.

15. Non-transitory computer readable storage media having instructions stored thereupon that, when executed by a processor of a system specially configured for diagnosing a Pulmonary Embolism (PE), the instructions cause the system to perform operations including:

receiving a plurality of medical images;

processing the plurality of medical images by executing an image-level classification algorithm to determine the presence or absence of Pulmonary Embolism (PE) within each image;

pre-training an AI model through supervised learning to identify ground truth;

fine-tuning the pre-trained AI model specifically for PE diagnosis to generate a pre-trained PE diagnosis and detection AI model;

wherein the pre-trained AI model is based on a modified CNN architecture having introduced therein a squeeze and excitation (SE) block enabling the CNN architecture to extract informative features from the plurality of medical images by fusing spatial and channel-wise information;

applying the pre-trained PE diagnosis and detection AI model to new medical images to render a prediction as to the presence or absence of a Pulmonary Embolism within the new medical images; and

outputting the prediction as a PE diagnosis for a medical patient.

16. The non-transitory computer readable storage media of claim 15 , wherein the new medical images constitute no part of any training set of medical images utilized to pre-train or fine-tune the AI model and have not been encountered by the pre-trained AI model prior to applying the pre-trained PE diagnosis and detection AI model to the new medical images.

17. The non-transitory computer readable storage media of claim 15 , wherein the instructions cause the system to perform operations further comprising:

pre-training a vision transformer (ViT) architecture with a training image dataset; and

upscaling each image in the training image dataset for a given patch size to increase the number of patches resulting in an enlarged size of the training image dataset.

18. The non-transitory computer readable storage media of claim 15 , wherein the instructions cause the system to perform operations further comprising:

pre-training an AI model using different sized images generated from the upscaling of each image in the training image dataset for a given patch size; and

wherein applying the pre-trained PE diagnosis and detection AI model to new medical images to render a prediction as to the presence or absence of a Pulmonary Embolism within the new medical images comprises applying an image-level classification procedure individually to each one of the new medical images to determine, for each one of the new medical images, either the Pulmonary Embolism is present within the respective one of the new medical images or the Pulmonary Embolism is absent from the respective one of the new medical images.

19. The non-transitory computer readable storage media of claim 15 , wherein pre-training an AI model through supervised learning to identify ground truth comprises applying pre-training via self-supervised learning using a publically available dataset having training images annotated as either PE presence or PE absence.

20. The non-transitory computer readable storage media of claim 15 , wherein receiving a plurality of medical images comprises executing instructions for receiving a training dataset having included therein the plurality of medical images; and

wherein the computer-implemented method further enlarges the training dataset's size beyond a received size for the training dataset by:

reshaping each of the plurality of images of the training dataset into a sequence of patches;

upscaling a corresponding image for each patch within the sequence of patches to generate an increased quantity of patches; and

wherein pre-training the AI model through supervised learning to identify ground truth comprises executing instructions for pre-training the AI model through supervised learning using the increased quantity of patches as an enlarged training dataset.

Assignments (2)
LICENSE Recorded May 10, 2024
From: ARIZONA STATE UNIVERSITY-TEMPE CAMPUS
To: NATIONAL INSTITUTES OF HEALTH
Reel/Frame 067381/0335 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 31, 2022
From: ISLAM, NAHID UL; GEHLOT, SHIV; ZHOU, ZONGWEI; LIANG, JIANMING
To: ARIZONA BOARD OF REGENTS ON BEHALF OF ARIZONA STATE UNIVERSITY
Reel/Frame 061596/0061 →
Continuity (2)
Provisional Application 63244183 · Sep 14, 2021
Related Publication 20230081305A1 · Mar 16, 2023
References Cited (60)
US 11547486B1 · Roh · 2023 [cited by examiner]
US 11672614B1 · Roh · 2023 [cited by examiner]
US 20220198662A1 · Ay · 2022 [cited by examiner]
US 20220208355A1 · Li · 2022 [cited by examiner]
Zbontar, J. et al., “Barlow Twins: Self-Supervised Learning via Redundancy Reduction,” International Conference on Machine Learning (PMLR), 2021, pp. 12310-12320. [cited by applicant]
Zhou, C., et al., “Computer-aided detection of pulmonary embolism in computed tomographic pulmonary angiography (CTPA): Performance evaluation with independent data sets,” Medical Physics, vol. 36, Issue 8, 2009, pp. 33… [cited by applicant]
Zhou, Z. et al., “Active, continual fine tuning of convolutional neural networks for reducing annotation efforts,” Medical Image Analysis, vol. 71, 2021, 101997, 15 pages. [cited by applicant]
Zhou, Z. et al., “Fine-tuning Convolutional Neural Networks for Biomedical Image Analysis: Actively and Incrementally,” In 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, pp. 4761-4772, 2017. [cited by applicant]
Zhou, Z. et al., “Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis,” Medical Image Computing and Computer Assisted Intervention-MICCAI 2019: 22nd International Conference, Shenzhen, China, Oct. … [cited by applicant]
Zhou, Z., “Towards Annotation-Efficient Deep Learning for Computer-Aided Diagnosis,” PhD thesis, Arizona State University, 2021, 214 pages. [cited by applicant]
Asano, Y.M. et al., “Self-labelling via simultaneous clustering and representation learning,” arXiv preprint arXiv:1911.05371v1 [cs.CV], 2019, 21 pages. [cited by applicant]
Asano, Y.M. et al., “Self-Labelling via Simultaneous Clustering and Representation Learning,” arXiv preprint arXiv:1911.05371v3 [cs. CV], 2020, 22 pages. [cited by applicant]
Brown, T.B., et al., “Language Models are Few-Shot Learnings,” Advances in Neural Information Processing Systems, 2020, pp. 1877-1901. [cited by applicant]
Brown, T.B., et al., “Language Models are Few-shot Learners,” arXiv preprint arXiv:2005.14165v4 [cs.CL], 2020, 75 pages. [cited by applicant]
Carbonneau, M.,A., et al. “Multiple Instance Learning: A Survey of Problem Characteristics and Applications,” arXiv preprint arXiv:1612.03365 (2016), 37 pages. [cited by applicant]
Caron, M. et al., “Deep Clustering for Unsupervised Learning of Visual Features,” In Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 132-149. [cited by applicant]
Caron, M. et al., “Unsupervised learning of visual features by contrasting cluster assignments,” arXiv preprint arXiv:2006.09882v5 [cs. CV], 2021, 23 pages. [cited by applicant]
Chen, T. et al., “A simple framework for contrastive learning of visual representations,” International Conference on Machine Learning, PMLR) 2020, pp. 1597-1607. [cited by applicant]
Chen, T. et al., “Big Self-Supervised Models are Strong Semi-supervised Learners,” Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 22243-22255. [cited by applicant]
Chen, X. et al., “Improved Baselines with Momentum Contrastive Learning,” arXiv preprint arXiv:2003.04297v1 [cs. CV], 2020, 3 pages. [cited by applicant]
Chollet, F., “Xception: Deep Learning with Depthwise Separable Convolutions,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1251-1258. [cited by applicant]
Colak, E. et al., “The RSNA Pulmonary Embolism CT Dataset,” Radiology: Artificial Intelligence, vol. 3, No. 2, 2021, 7 pages. [cited by applicant]
Deng, S. et al., “Deep learning in digital pathology image analysis: a survey,” Frontiers of Medicine, vol. 14, No. 1, 2020, pp. 470-487. [cited by applicant]
Devlin, J. et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” arXiv preprint arXiv: 1810.04805v1 [cs.CL], 2018, 16 pages. [cited by applicant]
Dosovitskiy, A. et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” arXiv preprint arXiv:2010.11929v2 [cs.CV] Jun. 3, 2021, 22 pages. [cited by applicant]
Gildenblat, J., et al., “Pytorch Library for Cam Methods,” 2021, https://github.com/jacobgil/pytorch-grad-cam, 11 pages. [cited by applicant]
Grill, J.B. et al., “Bootstrap Your Own Latent: A New Approach to Self-supervised Learning,” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 21271-21284. [cited by applicant]
Gutmann, M. et al., “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, JMLR … [cited by applicant]
Haghighi, F. et al., “Learning Semantics-enriched Representation via Self-discovery, Self-classification, and Self-restoration,” Medical Image Computing and Computer Assisted Intervention-MICCAI 2020, Lecture Notes in C… [cited by applicant]
Haghighi, F. et al., “Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learning,” IEEE Transactions on Medical Imaging, vol. 40, No. 10, 2021, pp. 2857-2868. [cited by applicant]
Han, K., et al., “Transformer in Transformer,” Advances in Neural Information Processing Systems, vol. 34, 2021, p. 15908-15919. [cited by applicant]
He, K. et al., “Deep Residual Learning for Image Recognition”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2016, pp. 770-778. [cited by applicant]
He, K. et al., “Momentum Contrast for Unsupervised Visual Representation Learning,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9729-9738. [cited by applicant]
Hu, D. et al., “How Well Does Self-Supervised Pre-training Perform with Streaming Data?,” arXiv preprint arXiv:2104.12081v3 [cs.LG], 2021, 23 pages. [cited by applicant]
Hu, J. et al., “Squeeze-and-Excitation Networks,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7132-7141. [cited by applicant]
Huang, G. et al., “Densely Connected Convolutional Networks,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2261-2269. [cited by applicant]
Huang, S.C. et al., “PENet—a scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging.” NPJ Digital Medicine vol. 3, No. 1, Article No. 61, 2020, 9 pages. [cited by applicant]
Ilse, M., et al., “Attention-based Deep Multiple Instance Learning,” International Conference on Machine Learning, Proceedings of Machine Learning Research, PLMR, 2018, pp. 2127-2136. [cited by applicant]
Islam, N.U., et al., “Seeking an Optimal Approach for Computer-Aided Pulmonary Embolism Detection,” 16 pages. [cited by applicant]
Jing, L., et al., “Self-Supervised Visual Feature Learning with Deep Neural Networks: A Survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, No. 11, 2021, pp. 4037-4058. [cited by applicant]
Leavitt, M.O., “Surgeon General's call to action to prevent deep vein thrombosis and pulmonary embolism,” 2009, DIANE Publishing. [cited by applicant]
Liang, J. et al., “Computer Aided Detection of Pulmonary Embolism with Tobogganing and Multiple Instance Classification in CT Pulmonary Angiography”, Biennial International Conference Information Processing in Medical I… [cited by applicant]
Litjens, G. et al., “A Survey on Deep Learning in Medical Image Analysis,” Medical Image Analysis, vol. 42, 2017, pp. 60-88. [cited by applicant]
Lucassen, W.A. et al. “Concerns in using multi-detector computed tomography for diagnosing pulmonary embolism in daily practice. A cross-sectional analysis using expert opinion as reference standard,” Thrombosis Researc… [cited by applicant]
Masutani, Y. et al., “Computerized Detection of Pulmonary Embolism in Spiral CT Angiography Based on Volumetric Image Analysis,” IEEE Transactions on Medical Imaging, vol. 21, No. 12, 2002, pp. 1517-1523. [cited by applicant]
Misra, I. et al. “Self-Supervised Learning of Pretext-Invariant Representations,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6707-6717. [cited by applicant]
Rajan, D. et al., “Pi-PE: A Pipeline for Pulmonary Embolism Detection using Sparsely Annotated 3D CT Images,” Proceedings of Machine Learning Research Machine Learning for Health, ML4H, 2020, pp. 220-232. [cited by applicant]
Shin, H.C. et al., “Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning,” IEEE Transactions on Medical Imaging, vol. 35, No. 5, 2016, pp. 128… [cited by applicant]
Stein, A., et al., “RSNA STR Pulmonary Embolism Detection,” https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/overview, 2020. Online; accessed Jun. 21, 2021. [cited by applicant]
Stein, A., et al., “RSNA STR Pulmonary Embolism Detection,” Kaggle https://www.kaggle.com/c/rsha-str-pulmonary-embolism-detection/discussion/194145 (2020), 11 pages. [cited by applicant]
Stein, P.D. et al., “Multidetector Computed Tomography for Acute Pulmonary Embolism,” The New England Journal of Medicine, vol. 354, No. 22, 2006, pp. 2317-2327. [cited by applicant]
Tajbakhsh, N. et al., “Computer-Aided Pulmonary Embolism Detection Using a Novel Vessel-aligned Multi-planar Image Representation and Convolutional Neural Networks,” Medical Image Computing and Computer-Assisted Interve… [cited by applicant]
Tajbakhsh, N. et al., “Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?,” IEEE Transactions on Medical Imaging, vol. 35, No. 5, May 2016, pp. 1299-1312. [cited by applicant]
Tian, Y. et al., “What Makes for Good Views for Contrastive Learning?,” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 6827-6839. [cited by applicant]
Touvron, H. et al., “Training data-efficient image transformers & distillation through attention,” Proceedings of the 38th International Conference on Machine Learning. PMLR, vol. 139, 2021, pp. 10347-10357. [cited by applicant]
U.S. Department of Health and Human Services Food and Drug Administration, “The Surgeon General's Call to Action to Prevent Deep Vein Thrombosis and Pulmonary Embolism,” vol. 119, Issue 15, 2009, pp. e480-e482. [cited by applicant]
Vaswani, A. et al., “Attention Is All You Need,” Advances in Neural Information Processing Systems, 2017, pp. 5998-6008. [cited by applicant]
Wu, Z. et al., “Unsupervised Feature Learning via Non-Parametric Instance Discrimination,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 3733-3742. [cited by applicant]
Xie, S. et al., “Aggregated Residual Transformations for Deep Neural Networks,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1492-1500. [cited by applicant]
Yu, F. et al., “Dilated Residual Networks,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 472-480. [cited by applicant]
Cited By (1)
US 12,511,747