IP Library › Granted Patent US 12,292,494
Granted Patent B2
US 12,292,494 · App. 17/264,620 · Granted May 6, 2025

Methods and systems for segmenting organs in images using a CNN-based correction network

Inventors: Deshan Yang (St. Louis, MO); Yabo Fu (St. Louis, MO)
Assignee: Washington University
G01R33/5608A61B5/201A61B5/4238A61B5/4244A61B5/4255A61B5/4836A61B5/7267G01R33/4808G06F18/2148G06F18/2185G06F18/2193G06N3/0455G06N3/0895G06N20/20G06T7/12G06V10/82G06T2207/10088G06T2207/20084G06T2207/30056G06T2207/30084G06T2207/30092G06V2201/031
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,292,494
App. No.
17/264,620
Granted
May 6, 2025
Kind
B2
Abstract

Among the various aspects of the present disclosure is the provision of methods and systems for segmenting images and expediting a contouring process for MRI-guided adaptive radiotherapy (MR-IGART) comprising applying a convolutional neural network (CNN), wherein the CNN accurately segments organs (e.g., the liver, kidneys, stomach, bowel, or duodenum) in 3D MR images.

Claims (26)

1. A method of segmenting contours of one or more abdominal cavity organs comprising:

(i) providing an image dataset;

(ii) applying a first convolutional neural network (sub-CNN1) to the image dataset, resulting in a label probability map of sub-CNN1 (P 1 ) (segmentation results);

(iii) applying a correction network comprising a second convolutional neural network (sub-CNN2) and a third convolutional neural network (sub-CNN3) to P 1 comprising:

(a) applying the second convolutional neural network (sub-CNN2) to the label probability map of sub-CNN1 (P 1 ), resulting in a label probability map of sub-CNN2 (P 2 );

(b) concatenating the image datasets, P 1 , and P 2 ; and

(c) applying the third convolutional neural network (sub-CNN3) to the image datasets, P 1 and P 2 ;

wherein applying the first convolutional neural network and the correction network results in accurately segmented organs in images; and

(iv) training sub-CNN1, sub-CNN2, and sub-CNN3, wherein training comprises:

training each of sub-CNN1, sub-CNN2, and sub-CNN3 parameters using whole 3D images to incorporate anatomical contextual information;

randomizing sub-CNN1, sub-CNN2, and sub-CNN3 parameters using a Gaussian distribution;

calculating a cross entropy loss function using a softmax classifier for sub-CNN1, wherein sub-CNN2 and sub-CNN3 are constant;

calculating a cross entropy loss function using a softmax classifier for sub-CNN2, wherein sub-CNN1 and sub-CNN3 are constant; and

calculating a cross entropy loss function using a softmax classifier for sub-CNN3, wherein sub-CNN1 and sub-CNN2 are constant.

2. The method of claim 1 , wherein one or more abdominal cavity organs are selected from gastrointestinal (GI) organs or organs at risk (OAR).

3. The method of claim 1 , wherein the one or more abdominal cavity organs are selected from one or both kidneys, a liver, a stomach, a bowel, or a duodenum.

4. The method of any one of claim 1 , wherein the one or more abdominal cavity organs is an unstable organ selected from one or more of the group consisting of: a stomach, a bowel, and a duodenum.

5. The method of claim 1 , wherein the label probability map of sub-CNN1 (P 1 ) (segmentation results) are refined through iterative error feedback comprising feeding erroneous labeling and the image dataset into the correction network, wherein the correction network iteratively corrects erroneous labels and improves segmentation accuracy.

6. The method of claim 1 , wherein the image dataset is a MR image dataset or a CT image dataset.

7. The method of claim 1 , wherein the correction network:

generates segmentations with possible organs shapes and locations;

iteratively improves the P 1 (the segmentation results);

is incorporated into the whole network and capable of being trained from end-to-end; and

performs robustly and does not require careful parameter fine tuning.

8. The method of claim 1 , wherein the correction network implicitly enforces anatomical constraints selected from the group consisting of: (i) a shape of one or more organs or (ii) relative locations of multiple organs.

9. An MRI-guided radiation treatment system comprising the use of the method of claim 1 .

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 5, 2021
From: YANG, DESHAN; FU, YABO
To: WASHINGTON UNIVERSITY
Reel/Frame 055162/0525 →
Continuity (3)
Provisional Application 62850225 · May 20, 2019
Provisional Application 62712619 · Jul 31, 2018
Related Publication 20210290096A1 · Sep 23, 2021
References Cited (67)
US 9589374B1 · Gao et al. · 2017 [cited by applicant]
US 9965863B2 · Xu et al. · 2018 [cited by applicant]
US 10147187B2 · Wang et al. · 2018 [cited by applicant]
US 10223610B1 · Akselrod-Ballin · 2019 [cited by examiner]
US 20080304616A1 · Van Uitert, Jr. · 2008 [cited by examiner]
US 20110212090A1 · Pedersen · 2011 [cited by examiner]
US 20170046616A1 · Socher · 2017 [cited by examiner]
US 20180157939A1 · Butt · 2018 [cited by examiner]
US 20180218502A1 · Golden et al. · 2018 [cited by applicant]
US 20180276815A1 · Xu et al. · 2018 [cited by applicant]
US 20180365888A1 · Satzoda · 2018 [cited by examiner]
US 20200019778A1 · Frei · 2020 [cited by examiner]
US 20200193299A1 · Geva · 2020 [cited by examiner]
US 20200279393A1 · Wolf · 2020 [cited by examiner]
US 20200380675A1 · Golden · 2020 [cited by examiner]
US 20210177373A1 · Xie · 2021 [cited by examiner]
CN 106204587A · 2016 [cited by applicant]
WO WO2018052587A1 · 2018 [cited by applicant]
Abiodun Oi et al. (2018) State-of-the-art in artificial neural network applications: A survey. Heliyon. vol. 4, No. e00938, 41 pages. [cited by applicant]
Anderson D & McNeill G (1992) Artificial Neural Networks Technology. DACS State-of-the-Art Report, Kaman Sciences Corporation. 87 pages. [cited by applicant]
Arjovsky M et al. (2017) Wasserstein generative adversarial networks. Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, pp. 214-223. [cited by applicant]
Bengio Y et al. (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Networks. vol. 5, No. 2, pp. 157-166. [cited by applicant]
Buda M et al. (2017) A systematic study of the class imbalance problem in convolutional neural networks. arXiv:1710.05381. 23 pages. [cited by applicant]
Chen L-C et al. (2016) Semantic image segmentation with deep convolutional nets and fully connected CRFs. ICLR 2015. arXiv:1412.7062v4. 14 pages. [cited by applicant]
Chen H et al. (2017) VoxResNet: deep voxelwise residual networks for brain segmentation from 3D Mr images. NeuroImage. vol. 170, pp. 446-455. [cited by applicant]
Chen L-C et al. (2018) DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans Pattern Anal Mach Intell. vol. 40, No. 4, pp. 834-848. [cited by applicant]
Christ PF et al. (2016) Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields. MICCAI 2016, Part II, LNCS 9901. pp. 415-423. [cited by applicant]
Christ PF et al. (2017) Automatic Liver and Tumor Segmentation of CT and MRI Volumes Using Cascaded Fully Convolutional Neural Networks. arXiv:1702.05970. 19 pages. [cited by applicant]
Çiçek Ö et al. (2016) 3D U-Net: learning dense volumetric segmentation from sparse annotation. MICCAI 2016, Part II, LNCS 9901. pp. 424-432. [cited by applicant]
Dai et al. (2017) Scan: Structure correcting adversarial network for organ segmentation in chest x-rays. DLMIA 2018/ML-CDS 2018, LNCS 11045, pp. 263-273. [cited by applicant]
Dolz J et al. (2017) 3D fully convolutional networks for subcortical segmentation in MRI: a large-scale study. NeuroImage. vol. 170, pp. 456-470. [cited by applicant]
Dosovitskiy A & Brox T (2016) Generating images with perceptual similarity metrics based on deep networks. 30th Conference on Neural Information Processing Systems (NIPS 2016). pp. 658-666. [cited by applicant]
Dou QP et al. (2017) 3D deeply supervised network for automated segmentation of volumetric medical images. Med Image Anal. vol. 41, pp. 40-54. [cited by applicant]
Fritscher K et al. (2016) Deep Neural networks for fast segmentation of 3D medical images. MICCAI 2016, Part II, LNCS 9901. pp. 158- 165. [cited by applicant]
Groza et al. (2018) Comparison of deep-learning based techniques for organ segmentation in abdominal CT images. 1 [cited by applicant]
Gulrajani I et al. (2017) Improved training of wasserstein GANs. Proceedings of the 31 [cited by applicant]
Hu et al. (2016) Automatic abdominal multi-organ segmentation using deep convolutional neural network and time-implicit level sets. Int J Comput Assist Radiol Surg, vol. 12, No. 3, pp. 399-411. [cited by applicant]
Hu P et al. (2016) Automatic 3D liver segmentation based on deep learning and globally optimized surface evolution. Phys Med Biol. vol. 61, pp. 8676-8698. [cited by applicant]
Huang G et al. (2017) Densely connected convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2261-2269. [cited by applicant]
Huttenlocher DP et al. (1993) Comparing images using the hausdorff distance. IEEE Trans Pattern Anal Mach Intell. vol. 15, No. 9, pp. 850-863. [cited by applicant]
International Search Report and Written Opinion for PCT/US2019/044118, mailed Nov. 1, 2019. 16 pages. [cited by applicant]
Ioffe S & Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. Proceedings of the 32 [cited by applicant]
Kamnitsas K et al. (2017) Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal. vol. 36, pp. 61-78. [cited by applicant]
Krähenbühl P & Koltun V (2011) Efficient inference in fully connected CRFs with gaussian edge potentials. Advances in Neural Information Processing Systems 24 (NIPS 2011). 9 pages. [cited by applicant]
Lamb J et al. (2017) Online adaptive radiation therapy: implementation of a new process of care. Cureus. vol. 9, No. 8, e1618, 8 pages. [cited by applicant]
Lin G et al. (2016) Efficient piecewise training of deep structured models for semantic segmentation. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3194-3203. [cited by applicant]
Litjens G et al. (2017) A survey on deep learning in medical image analysis. Med Image Anal. vol. 42, pp. 60-88. [cited by applicant]
Lu F et al. (2016) Automatic 3D liver location and segmentation via convolutional neural network and graph cut. arXiv:1605.03012. 12 pages. [cited by applicant]
Mescheder LM et al. (2018) Which training methods for GANs do actually Converge? Proceedings of the 35 [cited by applicant]
Milletari F et al. (2016) V-Net: fully convolutional neural networks for volumetric medical image segmentation. 2016 Fourth International Conference on 3D Vision (3DV), pp. 565-571. [cited by applicant]
Moeskops P et al. (2016) Automatic segmentation of MR brain images with a convolutional neural network. IEEE Trans Med Imaging. vol. 35, No. 5, pp. 1252-1262. [cited by applicant]
Moeskops P et al. (2016) Deep learning for multi-task medical image segmentation in multiple modalities. MICCAI 2016, Part II, LNCS 9901. pp. 478-486. [cited by applicant]
Mutic S & Dempsey JF (2014) The ViewRay system: magnetic resonance-guided and controlled radiotherapy. Semin Radiat Oncol. vol. 24, No. 3, pp. 196-199. [cited by applicant]
Noel CE et al. (2014) Segmentation precision of abdominal anatomy for MRI-based radiotherapy. Med Dosim. vol. 39, No. 3, pp. 212-217. [cited by applicant]
Ronneberger O et al. (2015) U-Net: convolutional networks for biomedical image segmentation. MICCAI 2015, Part III, LNCS 9351. pp. 234-241. [cited by applicant]
Roth et al. (2017) Hierarchical 3D fully convolutional networks for multi-organ segmentation. arXiv:1704.06382. 11 pages. [cited by applicant]
Roth K et al. (2017) Stabilizing training of generative adversarial networks through regularization. 31 [cited by applicant]
Shelhamer E et al. (2017) Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell. vol. 39, No. 4, pp. 640-651. [cited by applicant]
Srivastava N et al. (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res. vol. 15, pp. 1929-1958. [cited by applicant]
Wang et al. (2017) Interactive medical image segmentation using deep learning with image-specific fine tuning. IEEE Trans Med Imaging. vol. 37, No. 7. pp. 1562-1573. [cited by applicant]
Wooten Ho et al. (2015) Quality of intensity modulated radiation therapy treatment plans using a (6)(0)Co magnetic resonance image guidance radiation therapy system. Int J Radiat Oncol Biol Phys. vol. 92, No. 4, pp. 771… [cited by applicant]
Wooten Ho et al. (2015) Benchmark IMRT evaluation of a Co-60 MRI-guided radiation therapy system. Radiother Oncol. vol. 114, pp. 402-405. [cited by applicant]
Xing F et al. (2016) An automatic learning-based framework for robust nucleus segmentation. IEEE Trans Med Imaging. vol. 35, No. 2, pp. 550-566. [cited by applicant]
Xue Y et al. (2018) SegAN: adversarial network with multi-scale L 1 loss for medical image segmentation. Neuroinformatics. vol. 16, No. 3-4. pp. 383-392. [cited by applicant]
Yu L et al. (2017) Automatic 3D cardiovascular mr segmentation with densely-connected volumetric ConvNets. MICCAI 2017, Part II, LNCS 10434. pp. 287-295. [cited by applicant]
Zhang Z et al. (2018) Translating and segmenting multimodal medical vols. with cycle- and shape-consistency generative adversarial network. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9242-9… [cited by applicant]
Zhou X et al. (2017) Deep learning of the sectional appearances of 3D CT images for anatomical structure segmentation based on an FCN voting method. Med Phys. vol. 44, No. 10, pp. 5221-5233. [cited by applicant]