IP Library › Granted Patent US 12,561,807
Granted Patent B2
US 12,561,807 · App. 18/166,128 · Granted Feb 24, 2026

Optimization of a deep learning model for performing a medical imaging analysis task

Inventors: Bibo Shi (Monmouth Junction, NJ); Ali Kamen (Skillman, NJ); Tongbai Meng (Ellicott City, MD); Mamadou Diallo (Plainsboro, NJ); Bin Lou (Princeton Junction, NJ); Robert Grimm (Nuremberg, DE); Heinrich von Busch (Uttenreuth, DE); Berthold Kiefer (Erlangen, DE)
Assignee: Siemens Healthineers AG
G06T7/0014G06F18/214G06F18/2411G06N20/00G06T7/0012G06T7/11G06V10/26G06V10/454G06V10/764G06V10/82G06T2207/10088G06T2207/20081G06T2207/20084G06T2207/30096
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,561,807
App. No.
18/166,128
Granted
Feb 24, 2026
Kind
B2
Abstract

Systems and methods are provided for optimizing a deep learning model. A multi-site dataset associated with different clinical sites and a deployment dataset associated with a deployment clinical site are received. A deep learning model is trained based on the multi-site dataset. The trained deep learning model is optimized based on the deployment dataset. The optimized trained deep learning model is output.

Claims (57)

1 . A method for optimizing a deep learning model comprising:

receiving 1) a multi-site dataset associated with different clinical sites and 2) a deployment dataset associated with a deployment clinical site, wherein the multi-site dataset comprises a first annotated dataset and a second annotated dataset, the second annotated dataset comprising a plurality of sub-datasets each associated with a respective clinical site of the different clinical sites;

training a deep learning model for performing a medical image analysis task based on the multi-site dataset, wherein training a deep learning model for performing a medical image analysis task based on the multi-site dataset comprises:

reordering the plurality of sub-datasets of the second annotated dataset based on a similarity between the first annotated dataset and each of the plurality of the sub-datasets; and

updating the deep learning model based on the reordered plurality of sub-datasets, wherein the deep learning model was pretrained based on the first annotated dataset;

optimizing the trained deep learning model for performing the medical image analysis task based on uncertainties associated with the deployment dataset; and

outputting the optimized trained deep learning model.

2 . The method of claim 1 , wherein updating the deep learning model based on the reordered plurality of sub-datasets comprises:

for each respective sub-dataset of the reordered plurality of sub-datasets:

performing a first comparison and a second comparison, wherein the first comparison is between 1) a performance the deep learning model updated with the respective sub-dataset and any prior sub-datasets from prior iterations and 2) a performance of the deep learning model pretrained based on the first annotated dataset, and wherein the second comparison is between a) a performance of the deep learning model updated with the respective sub-dataset and any prior sub-datasets from prior iterations and b) a performance of the deep learning model trained on the first annotated dataset and the second annotated dataset; and

determining whether the deep learning model should be updated with the respective sub-dataset based on the first comparison and the second comparison and, in response to determining that the deep learning model should be updated with the respective sub-dataset, updating the deep learning model with the respective sub-dataset.

3 . The method of claim 2 , wherein determining whether the deep learning model should be updated with the respective sub-dataset based on the first comparison and the second comparison comprises:

determining that the deep learning model should be updated with the respective sub-dataset in response to determining that A) the performance of the deep learning model updated with the respective sub-dataset and any prior sub-datasets from prior iterations is greater than or equal to the performance of the deep learning model pretrained based on the first annotated dataset in the first comparison and B) the performance the deep learning model updated with the respective sub-dataset and any prior sub-datasets from prior iterations is greater than or equal to the performance of the deep learning model trained on the first annotated dataset and the second annotated dataset in the second comparison.

4 . The method of claim 1 , wherein the deployment dataset comprises an annotated deployment dataset and wherein optimizing the trained deep learning model for performing the medical image analysis task based on uncertainties associated with the deployment dataset comprises:

reordering the annotated deployment dataset based on the uncertainties; and

optimizing the trained deep learning model based on the reordered annotated deployment dataset.

5 . The method of claim 4 , wherein reordering the annotated deployment dataset based on the uncertainties comprises:

applying the trained deep learning model to each sample in the annotated deployment dataset to generate a respective pseudo-label; and

comparing an annotation associated with each sample in the annotated deployment dataset with its respective pseudo-label to determine an uncertainty associated with each sample in the annotated deployment dataset.

6 . The method of claim 4 , wherein optimizing the trained deep learning model based on the reordered annotated deployment dataset comprises:

for each respective sample in the reordered annotated deployment dataset:

performing a first comparison and a second comparison, wherein the first comparison is between 1) a performance of the trained deep learning model updated with the respective sample and any prior sample from prior iterations and 2) a performance of the trained deep learning mode, and wherein the second comparison is between a) a performance of the trained deep learning model updated with the respective sample and any prior sample from prior iterations and b) a performance of a deep learning model trained only with the annotated deployment dataset, and

determining whether to update the trained deep learning model with the respective sample based on the first comparison and the second comparison and, in response to determining that the trained deep learning model should be updated with the respective sample, updating the trained deep learning model with the respective sample.

7 . The method of claim 6 , wherein determining whether to update the trained deep learning model with the respective sample based on the first comparison and the second comparison comprises:

determining that the trained deep learning model should be updated with the respective sample in response to determining that A) the performance of the trained deep learning model updated with the respective sample and any prior sample from prior iterations is greater than or equal to the performance of the trained deep learning mode minus an error in the first comparison and B) the performance of the trained deep learning model updated with the respective sample and any prior sample from prior iterations is greater than or equal to the performance of a deep learning model trained only with the annotated deployment dataset in the second comparison.

8 . An apparatus for optimizing a deep learning model comprising:

means for receiving 1) a multi-site dataset associated with different clinical sites and 2) a deployment dataset associated with a deployment clinical site, wherein the multi-site dataset comprises a first annotated dataset and a second annotated dataset, the second annotated dataset comprising a plurality of sub-datasets each associated with a respective clinical site of the different clinical sites;

means for training a deep learning model for performing a medical image analysis task based on the multi-site dataset, wherein the means for training a deep learning model for performing a medical image analysis task based on the multi-site dataset comprises:

means for reordering the plurality of sub-datasets of the second annotated dataset based on a similarity between the first annotated dataset and each of the plurality of the sub-datasets; and

means for updating the deep learning model based on the reordered plurality of sub-datasets, wherein the deep learning model was pretrained based on the first annotated dataset;

means for optimizing the trained deep learning model for performing the medical image analysis task based on uncertainties associated with the deployment dataset; and

means for outputting the optimized trained deep learning model.

9 . The apparatus of claim 8 , wherein the means for updating the deep learning model based on the reordered plurality of sub-datasets comprises:

for each respective sub-dataset of the reordered plurality of sub-datasets:

means for performing a first comparison and a second comparison, wherein the first comparison is between 1) a performance the deep learning model updated with the respective sub-dataset and any prior sub-datasets from prior iterations and 2) a performance of the deep learning model pretrained based on the first annotated dataset, and wherein the second comparison is between a) a performance of the deep learning model updated with the respective sub-dataset and any prior sub-datasets from prior iterations and b) a performance of the deep learning model trained on the first annotated dataset and the second annotated dataset; and

means for determining whether the deep learning model should be updated with the respective sub-dataset based on the first comparison and the second comparison and, in response to determining that the deep learning model should be updated with the respective sub-dataset, updating the deep learning model with the respective sub-dataset.

10 . The apparatus of claim 9 , wherein the means for determining whether the deep learning model should be updated with the respective sub-dataset based on the first comparison and the second comparison comprises:

means for determining that the deep learning model should be updated with the respective sub-dataset in response to determining that A) the performance of the deep learning model updated with the respective sub-dataset and any prior sub-datasets from prior iterations is greater than or equal to the performance of the deep learning model pretrained based on the first annotated dataset in the first comparison and B) the performance the deep learning model updated with the respective sub-dataset and any prior sub-datasets from prior iterations is greater than or equal to the performance of the deep learning model trained on the first annotated dataset and the second annotated dataset in the second comparison.

11 . A non-transitory computer readable medium storing computer program instructions for optimizing a deep learning model, the computer program instructions when executed by a processor cause the processor to perform operations comprising:

receiving 1) a multi-site dataset associated with different clinical sites and 2) a deployment dataset associated with a deployment clinical site, wherein the multi-site dataset comprises a first annotated dataset and a second annotated dataset, the second annotated dataset comprising a plurality of sub-datasets each associated with a respective clinical site of the different clinical sites;

training a deep learning model for performing a medical image analysis task based on the multi-site dataset, wherein training a deep learning model for performing a medical image analysis task based on the multi-site dataset comprises:

reordering the plurality of sub-datasets of the second annotated dataset based on a similarity between the first annotated dataset and each of the plurality of the sub-datasets; and

updating the deep learning model based on the reordered plurality of sub-datasets, wherein the deep learning model was pretrained based on the first annotated dataset;

optimizing the trained deep learning model for performing the medical image analysis task based on uncertainties associated with the deployment dataset; and

outputting the optimized trained deep learning model.

12 . The non-transitory computer readable medium of claim 11 , wherein the deployment dataset comprises an annotated deployment dataset and wherein optimizing the trained deep learning model for performing the medical image analysis task based on uncertainties associated with the deployment dataset comprises:

reordering the annotated deployment dataset based on the uncertainties; and

optimizing the trained deep learning model based on the reordered annotated deployment dataset.

13 . The non-transitory computer readable medium of claim 12 , wherein reordering the annotated deployment dataset based on the uncertainties comprises:

applying the trained deep learning model to each sample in the annotated deployment dataset to generate a respective pseudo-label; and

comparing an annotation associated with each sample in the annotated deployment dataset with its respective pseudo-label to determine an uncertainty associated with each sample in the annotated deployment dataset.

14 . The non-transitory computer readable medium of claim 12 , wherein optimizing the trained deep learning model based on the reordered annotated deployment dataset comprises:

for each respective sample in the reordered annotated deployment dataset:

performing a first comparison and a second comparison, wherein the first comparison is between 1) a performance of the trained deep learning model updated with the respective sample and any prior sample from prior iterations and 2) a performance of the trained deep learning mode, and wherein the second comparison is between a) a performance of the trained deep learning model updated with the respective sample and any prior sample from prior iterations and b) a performance of a deep learning model trained only with the annotated deployment dataset, and

determining whether to update the trained deep learning model with the respective sample based on the first comparison and the second comparison and, in response to determining that the trained deep learning model should be updated with the respective sample, updating the trained deep learning model with the respective sample.

15 . The non-transitory computer readable medium of claim 14 , wherein determining whether to update the trained deep learning model with the respective sample based on the first comparison and the second comparison comprises:

determining that the trained deep learning model should be updated with the respective sample in response to determining that A) the performance of the trained deep learning model updated with the respective sample and any prior sample from prior iterations is greater than or equal to the performance of the trained deep learning mode minus an error in the first comparison and B) the performance of the trained deep learning model updated with the respective sample and any prior sample from prior iterations is greater than or equal to the performance of a deep learning model trained only with the annotated deployment dataset in the second comparison.

Assignments (4)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 20, 2023
From: SIEMENS HEALTHCARE GMBH
To: SIEMENS HEALTHINEERS AG
Reel/Frame 066267/0346 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 9, 2023
From: SHI, BIBO; KAMEN, ALI; DIALLO, MAMADOU; LOU, BIN; MENG, TONGBAI
To: SIEMENS MEDICAL SOLUTIONS USA, INC.
Reel/Frame 062641/0105 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 9, 2023
From: GRIMM, ROBERT; KIEFER, BERTHOLD; VON BUSCH, HEINRICH
To: SIEMENS HEALTHCARE GMBH
Reel/Frame 062641/0164 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 9, 2023
From: SIEMENS MEDICAL SOLUTIONS USA, INC.
To: SIEMENS HEALTHCARE GMBH
Reel/Frame 062641/0217 →
Continuity (5)
Continuation 17809385 · Jun 28, 2022
Continuation 15733778
Provisional Application 62687294 · Jun 20, 2018
Provisional Application 62684337 · Jun 13, 2018
Related Publication 20230267611A1 · Aug 24, 2023
References Cited (60)
US 8204315B2 · Madabhushi · 2012 [cited by examiner]
US 10424087B2 · Risser et al. · 2019 [cited by applicant]
US 10445910B2 · Tanaka et al. · 2019 [cited by applicant]
US 11295165B1 · Agarwal · 2022 [cited by examiner]
US 20160232425A1 · Huang · 2016 [cited by examiner]
US 20160239969A1 · Davatzikos et al. · 2016 [cited by applicant]
US 20170333521A1 · Hulme · 2017 [cited by examiner]
US 20180144465A1 · Hsieh · 2018 [cited by examiner]
US 20180144466A1 · Hsieh · 2018 [cited by examiner]
US 20180240233A1 · Kiraly · 2018 [cited by examiner]
US 20190156477A1 · Perrin · 2019 [cited by examiner]
US 20190266436A1 · Prakash · 2019 [cited by examiner]
US 20190287283A1 · Lin et al. · 2019 [cited by applicant]
US 20190304092A1 · Akselrod-Ballin et al. · 2019 [cited by applicant]
US 20190340470A1 · Hsieh · 2019 [cited by examiner]
US 20190355102A1 · Lin et al. · 2019 [cited by applicant]
US 20210217524A1 · Vos et al. · 2021 [cited by applicant]
US 20210248736A1 · Kamen · 2021 [cited by examiner]
US 20210249142A1 · Lau et al. · 2021 [cited by applicant]
Siegel et al., “Cancer Statistics, 2017”, CA: A Cancer Journal for Clinicians, vol. 67, No. 1, Jan./Feb. 2017, pp. 7-30. [cited by applicant]
Chung et al., “Discovery Radiomics for Multi-Parametric MRI Prostate Cancer Detection”, Computer Vision and Pattern Recognition, Oct. 20, 2015, 8 pgs. [cited by applicant]
Reda et al., “Computer-Aided Diagnostic Tool for Early Detection of Prostate Cancer”, IEEE International Conference on Image Processing, Sep. 2016. [cited by applicant]
Wang et al., “Computer Aided-Diagnosis of Prostate Cancer on Multiparametric MRI: A Technical Review of Current Research”, BioMed Research International, vol. 2014, 11 pages. [cited by applicant]
Wang et al., “Automated Detection of Clinically Significant Prostate Cancer in mp-MRI Images Based on an End-to-End Deep Neural Network”, IEEE Transactions on Medical Imaging, vol. 37, Issue 5, May 2018, pp. 1127-1139. [cited by applicant]
Lin et al., “Bilinear CNN Models for Fine-Grained Visual Recognition”, Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1449-1457. [cited by applicant]
Havaei et al., “HeMIS: Hetero-Modal Image Segmentation”, International Conference on Medical Image Computing and Computing and Computer-Assisted Intervention, Jul. 18, 2016, 11 pgs. [cited by applicant]
He et al., “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification”, Proceedings of the IEEE International Conference on Computer Vision, Feb. 6, 2015, 11 pgs. [cited by applicant]
Litjens et al., “A Survey on Deep Learning in Medical Image Analysis”, Medical Image Analysis 42, 2017, pp. 50-88. [cited by applicant]
Shen et al., “Deep Learning in Medical Image Analysis”, Annual Review of Biodmedical Engineering, Jun. 21, J017, pp. 221-248. [cited by applicant]
Shin et al., “Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning”, IEEE Transactions on Medical Imaging, vol. 35, No. 5, May 2016, pp. 1285-… [cited by applicant]
Heckman et al., “Sample Selection Bias as a Specification Error”, Econometrica, vol. 47, No. 1, 1979, pp. 153-161. [cited by applicant]
Mazzara et al., “Brain Tumor Target vol. Determination for Radiation Treatment Planning Through Automated MRI Segmentation”, International Journal of Radiation Oncology, Physics, vol. 59, No. 1, 2004, pp. 300-312. [cited by applicant]
Kouw et al., “A Review of Single-Source Unsupervised Domain Adaption”, arXiv:1901.05335v1; Jan. 16, 2019, 18 pgs. [cited by applicant]
Wang et al., “Deep Visual Domain Adaptation: A Survey”, Neurocomputing, 2018, 17 pgs. [cited by applicant]
Sun et al., “A Survey of Multi-Source Domain Adaption”, Information Fusion 24, 2015, pp. 84-92. [cited by applicant]
Patel et al., “Visual Domain Adaption: A Survey of Recent Advances”, IEEE Signal Processing Magazine, vol. 32, Issue 3, 2015, pp. 53-69. [cited by applicant]
Van Opbroek, et al., “Transfer Learning Improves Supervised Image Segmentation Across Imaging Protocols”, IEEE Transactions on Medical Imaging, Nov. 2014, 14 pgs. [cited by applicant]
Tajbakhsh et al., “Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning”, IEEE Transactions on Medical Imaging, vol. 35, Issue 5, 2016, pp. 1299-1312. [cited by applicant]
Ghafoorian et al., “Transfer Learning for Domain Adaptation in MRI: Application in Brain Lesion Segmentation”, International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer Cham, Feb. … [cited by applicant]
Valindria et al., “Domain Adaption for MRI Organ Segmentation using Reverse Classification Accuracy”, rXiv:1806.00363v1, Jun. 1, 2018, 9 pgs. [cited by applicant]
Mahmood et al., “Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training”, IEEE Transactions on Medical Imaging, Nov. 29, 2017, 10 pgs. [cited by applicant]
Bengio et al., “Curriculum Leaming”, Proceedings of the 26th Annual International Conference on Machine Learning, ACM, 2009, 8 pgs. [cited by applicant]
Pi et al., “Self-Paced Boost Learning for Classification”, Proceedings of the 25th International Joint Conference on Artificial Intelligence, 2016, 7 pgs. [cited by applicant]
Lambin et al., “Radiomics: Extracting More Information from Medical Images Using Advanced Feature Analysis”, European Journal of Cancer, 2012, pp. 441-446. [cited by applicant]
Pearson, “On lines and planes of closest fit to systems of points in space”, Jun. 8, 2010, The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2:11, 559-572, DOI: 10.1080/14786440109462720. [cited by applicant]
Fisher, “The Use of Multiple Measurements in Taxonomic Problems”, Annals of Eugenics, 1936, 10 pgs. [cited by applicant]
Scholkopf, “Nonlinear Component Analysis as a Kernel Eigenvalue Problem”, Neural Computation, 1998, 21 pgs. [cited by applicant]
Baudat et al., “Generalized Discriminant Analysis Using a Kernel Approach”, Neural Computation, vol. 12, Issue 10, Oct. 2000, pp. 2385-2404. [cited by applicant]
Tenenbaum et al., “A Global Geometric Framework for Nonlinear Dimensionality Reduction”, Science, vol. 290, Dec. 22, 2000, pp. 2319-2323. [cited by applicant]
Roweis et al., “Nonlinear Dimensionality Reduction by Locally Linear Embedding”, Science, vol. 290, Dec. 22, 2000, pp. 2323-2326. [cited by applicant]
Hinton et al., “A Fast Learning Algorithm for Deep Belief Nets”, Neural Computation 18, 2006, pp. 1527-1554. [cited by applicant]
Hinton et al., “Reducing the Dimensionality of Data with Neural Networks”, Science, Jul. 28, 2006, vol. 313, Issue 5786, pp. 504-507. [cited by applicant]
McCulloch et al., “A Logical Calculus of the Ideas Immanent in Nervous Activity”, Bulletin of Mathematical Biphysics, vol. 5, 1943, pp. 115-133. [cited by applicant]
Zhong et al., “An Overview on Data Representation Learning: From Traditional Feature Learning to Recent Deep Learning”, Journal of Finance and Data Science, Nov. 25, 2016, 21 pgs. [cited by applicant]
Efros et al., “Texture Synthesis by Non-Parametric Sampling”, 1999, IEEE, 6 pgs. [cited by applicant]
Xu et al., “Image Inpainting by Patch Propagation Using Patch Sparsity”, IEEE Transactions on Image Processing, vol. 19, No. 5, May 2010, pp. 1153-1165. [cited by applicant]
Radford et al., “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks”, ICLR, Nov. 19, 2015, 15 pgs. [cited by applicant]
Sachin Mehta et al., “Y-Net: Joint Segmentation and Classification for Diagnosis of Breast Biopsy Images”, Jun. 4 J018, pp. 1-9. [cited by applicant]
International Search Report mailed Nov. 25, 2019 in connection with International Patent Application No. PCT/EP2019/065447. [cited by applicant]
Chen, et al.; “Fully automatic acute ischemic lesion segmentation in DWI using convolutional neural networks”, NeuroImage: Clinical, vol. 15, Jan. 2017, pp. 633-643. [cited by applicant]