IP Library › Granted Patent US 12,394,185
Granted Patent B2
US 12,394,185 · App. 18/058,884 · Granted Aug 19, 2025

Cross domain segmentation with uncertainty-guided curriculum learning

Inventors: Yue Zhang (Jersey City, NJ); Caius Constantin Suliman (Brasov, RO); Florin-Cristian Ghesu (Baiersdorf, DE); Rui Liao (Princeton Junction, NJ)
Assignee: Siemens Healthineers AG
G06V10/774G06T7/0012G06V10/26G06T2207/30168G06V2201/03
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,394,185
App. No.
18/058,884
Granted
Aug 19, 2025
Kind
B2
Abstract

Systems and methods for training a machine learning based segmentation network are provided. A set of medical images, each depicting an anatomical object, in a first modality is received. For each respective medical image of the set of medical images, a synthetic image, depicting the anatomical object, in a second modality is generated based on the respective medical image. One or more augmented images are generated based on the synthetic image. One or more segmentations of the anatomical object are performed from the one or more augmented images using a machine learning based reference network. An uncertainty associated with segmenting the anatomical object from the respective medical image is computed based on results of the one or more segmentations. It is determined whether the respective medical image is suitable for training a machine learning based segmentation network based on the uncertainty. The machine learning based segmentation network is trained based on 1) the suitable medical images of the set of medical images and 2) annotations of the anatomical object determined using a machine learning based teacher network.

Claims (84)

1. A method comprising:

receiving a set of medical images, each depicting an anatomical object, in a first modality;

for each respective medical image of the set of medical images:

generating a synthetic image, depicting the anatomical object, in a second modality based on the respective medical image;

generating one or more augmented images based on the synthetic image;

performing one or more segmentations of the anatomical object from the one or more augmented images using a machine learning based reference network;

computing an uncertainty associated with segmenting the anatomical object from the respective medical image by:

quantifying a quality of each of the one or more segmentations of the anatomical object from the one or more augmented images,

calculating an average quality based on the quality of each of the one or more segmentations; and

computing the uncertainty based on a distance between the quality of each of the one or more segmentations and the average quality; and

determining whether the respective medical image is suitable for training a machine learning based segmentation network based on the uncertainty; and

training the machine learning based segmentation network based on 1) the suitable medical images of the set of medical images and 2) annotations of the anatomical object determined using a machine learning based teacher network.

2. The method of claim 1 , wherein determining whether the respective medical image is suitable for training a machine learning based segmentation network based on the uncertainty comprises:

comparing the uncertainty with a threshold.

3. The method of claim 2 , further comprising:

repeating the generating, the performing, the computing, the determining, and the training for a plurality of epochs.

4. The method of claim 3 , wherein the threshold is updated after each of the plurality of epochs.

5. The method of claim 3 , further comprising:

in response to determining that the respective medical image is suitable for training the machine learning based segmentation network, updating a counter representing a frequency that the respective medical image has been determined as being suitable for training the machine learning based segmentation network.

6. The method of claim 5 , further comprising, during a next epoch:

determining whether the respective medical image is suitable for training the machine learning based segmentation network based on the counter; and

training the machine learning based segmentation network based on the respective medical image determined to be suitable for training the machine learning based segmentation network based on the counter.

7. The method of claim 1 , wherein generating one or more augmented images based on the synthetic image comprises:

applying one or more transformations to the synthetic image to generate the one or more augmented images.

8. The method of claim 1 , wherein computing the uncertainty based on a distance between the quality of each of the one or more segmentations and the average quality comprises:

computing a maximum deviation of the distance between the quality of each of the one or more segmentations and the average quality.

9. The method of claim 1 , wherein training the machine learning based segmentation network based on 1) the suitable medical images of the set of medical images and 2) annotations of the anatomical object determined using a machine learning based teacher network comprises:

training the machine learning based segmentation network based on one or more additional annotated medical images.

10. An apparatus comprising:

means for receiving a set of medical images, each depicting an anatomical object, in a first modality;

for each respective medical image of the set of medical images:

means for generating a synthetic image, depicting the anatomical object, in a second modality based on the respective medical image;

means for generating one or more augmented images based on the synthetic image;

means for performing one or more segmentations of the anatomical object from the one or more augmented images using a machine learning based reference network;

means for computing an uncertainty associated with segmenting the anatomical object from the respective medical image by:

quantifying a quality of each of the one or more segmentations of the anatomical object from the one or more augmented images,

calculating an average quality based on the quality of each of the one or more segmentations; and

computing the uncertainty based on a distance between the quality of each of the one or more segmentations and the average quality; and

means for determining whether the respective medical image is suitable for training a machine learning based segmentation network based on the uncertainty; and

means for training the machine learning based segmentation network based on 1) the suitable medical images of the set of medical images and 2) annotations of the anatomical object determined using a machine learning based teacher network.

11. The apparatus of claim 10 , wherein the means for determining whether the respective medical image is suitable for training a machine learning based segmentation network based on the uncertainty comprises:

means for comparing the uncertainty with a threshold.

12. The apparatus of claim 11 , further comprising:

means for repeating the generating, the performing, the computing, the determining, and the training for a plurality of epochs.

13. The apparatus of claim 12 , wherein the threshold is updated after each of the plurality of epochs.

14. The apparatus of claim 12 , further comprising:

in response to determining that the respective medical image is suitable for training the machine learning based segmentation network, means for updating a counter representing a frequency that the respective medical image has been determined as being suitable for training the machine learning based segmentation network.

15. The apparatus of claim 14 , further comprising, during a next epoch:

means for determining whether the respective medical image is suitable for training the machine learning based segmentation network based on the counter; and

means for training the machine learning based segmentation network based on the respective medical image determined to be suitable for training the machine learning based segmentation network based on the counter.

16. A non-transitory computer readable medium storing computer program instructions, the computer program instructions when executed by a processor cause the processor to perform operations comprising:

receiving a set of medical images, each depicting an anatomical object, in a first modality;

for each respective medical image of the set of medical images:

generating a synthetic image, depicting the anatomical object, in a second modality based on the respective medical image;

generating one or more augmented images based on the synthetic image;

performing one or more segmentations of the anatomical object from the one or more augmented images using a machine learning based reference network;

computing an uncertainty associated with segmenting the anatomical object from the respective medical image by:

quantifying a quality of each of the one or more segmentations of the anatomical object from the one or more augmented images,

calculating an average quality based on the quality of each of the one or more segmentations; and

computing the uncertainty based on a distance between the quality of each of the one or more segmentations and the average quality; and

determining whether the respective medical image is suitable for training a machine learning based segmentation network based on the uncertainty; and

training the machine learning based segmentation network based on 1) the suitable medical images of the set of medical images and 2) annotations of the anatomical object determined using a machine learning based teacher network.

17. The non-transitory computer readable medium of claim 16 , wherein generating one or more augmented images based on the synthetic image comprises:

applying one or more transformations to the synthetic image to generate the one or more augmented images.

18. The non-transitory computer readable medium of claim 16 , wherein computing the uncertainty based on a distance between the quality of each of the one or more segmentations and the average quality comprises:

computing a maximum deviation of the distance between the quality of each of the one or more segmentations and the average quality.

19. The non-transitory computer readable medium of claim 16 , wherein training the machine learning based segmentation network based on 1) the suitable medical images of the set of medical images and 2) annotations of the anatomical object determined using a machine learning based teacher network comprises:

training the machine learning based segmentation network based on one or more additional annotated medical images.

20. A method comprising:

receiving an input medical image depicting an anatomical object;

performing a segmentation of the anatomical object from the input medical image using a trained machine learning based segmentation network; and

outputting results of the segmentation,

wherein the trained machine learning based segmentation network is trained by:

receiving a set of medical images, each depicting the anatomical object, in a first modality;

for each respective medical image of the set of medical images:

generating a synthetic image, depicting the anatomical object, in a second modality based on the respective medical image;

generating one or more augmented images based on the synthetic image;

performing one or more segmentations of the anatomical object from the one or more augmented images using a machine learning based reference network;

computing an uncertainty associated with segmenting the anatomical object from the respective medical image by:

quantifying a quality of each of the one or more segmentations of the anatomical object from the one or more augmented images,

calculating an average quality based on the quality of each of the one or more segmentations; and

computing the uncertainty based on a distance between the quality of each of the one or more segmentations and the average quality; and

determining whether the respective medical image is suitable for training a machine learning based segmentation network based on the uncertainty; and

training the machine learning based segmentation network based on 1) the suitable medical images of the set of medical images and 2) annotations of the anatomical object determined using a machine learning based teacher network.

Assignments (6)
CORRECTIVE ASSIGNMENT TO CORRECT THE APPLICATION NO. 15058884 PREVIOUSLY RECORDED AT REEL: 62008 FRAME: 847. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT . Recorded Jul 24, 2024
From: SIEMENS S.R.L.
To: SIEMENS MEDICAL SOLUTIONS USA, INC.
Reel/Frame 068352/0227 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 20, 2023
From: SIEMENS HEALTHCARE GMBH
To: SIEMENS HEALTHINEERS AG
Reel/Frame 066267/0346 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 12, 2022
From: SIEMENS MEDICAL SOLUTIONS USA, INC.
To: SIEMENS HEALTHCARE GMBH
Reel/Frame 062049/0265 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 7, 2022
From: SULIMAN, CAIUS CONSTANTIN
To: SIEMENS S.R.L.
Reel/Frame 062004/0009 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 7, 2022
From: SIEMENS S.R.L.
To: SIEMENS MEDICAL SOLUTIONS USA, INC.
Reel/Frame 062008/0847 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 7, 2022
From: ZHANG, YUE; GHESU, FLORIN-CRISTIAN; LIAO, RUI
To: SIEMENS MEDICAL SOLUTIONS USA, INC.
Reel/Frame 062013/0776 →
Continuity (1)
Related Publication 20240177458A1 · May 30, 2024
References Cited (45)
US 20190147298A1 · Rabinovich · 2019 [cited by examiner]
US 20200160177A1 · Durand · 2020 [cited by examiner]
US 20210150710A1 · Hosseinzadeh Taher · 2021 [cited by examiner]
US 20220114444A1 · Weinzaepfel · 2022 [cited by examiner]
US 20230046321A1 · Vilsmeier · 2023 [cited by examiner]
US 20230169332A1 · Karthik · 2023 [cited by examiner]
US 20230267611A1 · Shi · 2023 [cited by examiner]
US 20230316544A1 · Amadou · 2023 [cited by examiner]
Tajbakhsh, Nima, et al. “Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation.” Medical image analysis 63:101693. (Year: 2020). [cited by examiner]
Yu, Lequan, et al. “Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation.” Medical image computing and computer assisted intervention—MICCAI 2019: 22nd international conference, Shenzh… [cited by examiner]
Extended European Search Report (EESR) mailed May 23, 2023 in corresponding European Patent Application No. 22209867.5. [cited by applicant]
Zhang, Zizhao, Lin Yang, and Yefeng Zheng. “Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network.” Proceedings of the IEEE Conference on Computer Vision a… [cited by applicant]
Yu Lequan et al: “Uncertainty-Aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentation”, Oct. 10, 2019, 16th European Conference—Computer Vision—ECCV 2020, Cornell University Library, pp. 605-613, XP0… [cited by applicant]
Tajbakhsh Nima et al: “Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation”, arxiv.org, Cornell University Library, Aug. 27, 2019, XP08159752. [cited by applicant]
Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale”, arXiv:2010.11929, 2020, pp. 1-21. [cited by applicant]
Suetens et al., “Fundamentals of Medical Imaging, Third Edition”, Cambridge University Press, 2017, pp. 8-14, 30, 45-46 and 58, ISBN 978-1-107-15978-5. [cited by applicant]
Loshchilov et al., “Decoupled weight decay regularization”, arXiv:1711.05101, 2017, pp. 1-19. [cited by applicant]
London Department of Health, NHS Improvement, “Diagnostic imaging dataset statistical release”, 2016, pp. 1-17. [cited by applicant]
Wang et al., “COVID-Net: A tailored deep convolutional neural network design for detection of COVID-19 cases from chest X-ray images”, Scientific Reports, 2020, 12 pgs. [cited by applicant]
Hoffman et al., “CyCADA: Cycle-consistent adversarial domain adaptation”, International Conference on Machine Learning, 2018, pp. 1-15. [cited by applicant]
Zhang et al., “Unsupervised X-ray image segmentation with task driven generative adversarial networks”, Medical Image Analysis, 2020, pp. 1-15. [cited by applicant]
Li et al., “Dual-teacher: Integrating intra-domain and inter-domain teachers for annotation-efficient cardiac segmentation”, arxIV:2007.06279v1, 2020, 10 pgs. [cited by applicant]
Zheng et al., “Pairwise domain adaptation module for CNN-based 2-D/3-D Registration”, Journal of Medical Imaging, 2018, pp. 021204-1-021204-10. [cited by applicant]
Chen et al., “Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation”, Proceedings of the AAAI Conference on Artificial Intelligence, 2019, pp. 865-872. [cited by applicant]
Xu et al., “Cross-site severity assessment of COVID-19 from CT images via Domain Adaptation”, IEEE Transactions on Medical Imaging, 2021, pp. 1-15. [cited by applicant]
Chang et al., “All about structure: Adapting structural information across domains for boosting semantic segmentation”, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1-10. [cited by applicant]
Cui et al., “Structure-driven unsupervised domain adaptation for cross-modality cardiac segmentation”, IEEE Transactions on Medical Imaging, 2021, pp. 3604-3616. [cited by applicant]
Salimans et al., “Improved techniques for training GANs”, Advances in Neural Information Processing Systems, 2016, 9 pgs. [cited by applicant]
Tarvainen et al., “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results”, Advances in Neural Information Processing Systems, 2017, pp. 1-16. [cited by applicant]
Laine et al., “Temporal ensembling for semi-supervised learning”, arXiv:1610.02242, 2016, pp. 1-13. [cited by applicant]
Chen et al., “Improved baselines with momentum contrastive learning”, arXiv:2003.04297, 2020, pp. 1-3. [cited by applicant]
Grill et al., “Bootstrap your own latent a new approach to self-supervised learning”, Advances in Neural Information Processing Systems, 2020, pp. 1-35. [cited by applicant]
Chen et al., “Exploring Simple Siamese Representation Learning”, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 1-10. [cited by applicant]
Ghesu et al., “Self-supervised learning from 100 million medical images”, arXiv:2201.01283, 2022, pp. 1-13. [cited by applicant]
Ruijters et al., “GPU-accelerated digitally reconstructed radiographs”, BioMED: Proceedings of the Sixth IASTED International Conference on Biomedical Engineering, 2008, 5 pgs. [cited by applicant]
Bengio et al., “Curriculum Learning”, Proceedings of the 26th Annual International Conference on Machine Learning, 2009, 8 pgs. [cited by applicant]
Arjovsky et al., “Towards principled methods for training generative adversarial networks”, arXiv:1701.04862, 2017, pp. 1-16. [cited by applicant]
Roth et al., “Stabilizing training of generative adversarial networks through regularization”, Advances in Neural Information Processing Systems, 2017, pp. 1-16. [cited by applicant]
Zhu et al., “Unpaired image-to-image translation using cycle-consistent adversarial networks”, Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1-25. [cited by applicant]
J'egou et al., “The one hundred layers tiramisu: Fully convolutional denseNets for semantic segmentation”, Computer Vision and Pattern Recognition Workshops, 2017, pp. 1-9. [cited by applicant]
Wang et al., “DiCyc: GAN-based deformation invariant cross-domain information fusion for medical image synthesis”, Information Fusion, 2021, pp. 147-160. [cited by applicant]
Cubuk et al., “RandAugment: Practical automated data augmentation with a reduced search space”, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 1-12. [cited by applicant]
Demner-Fushman et al., “Preparing a collection of radiology examinations for distribution and retrieval”, Journal of the American Medical Informatics Association, 2016, pp. 304-310. [cited by applicant]
Shiraishi et al., “Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists' detection of pulmonary nodules”, American Jour… [cited by applicant]
You et al., “Towards accurate model selection in deep unsupervised domain adaptation”, International Conference on Machine Learning, PMLR, 2019, 10 pgs. [cited by applicant]