IP Library › Granted Patent US 12,228,629
Granted Patent B2
US 12,228,629 · App. 17/496,104 · Granted Feb 18, 2025

Deep learning methods for noise suppression in medical imaging

Inventors: Neel Dey (Brooklyn, NY); Jo Schlemper (Long Island City, NY); Seyed Sadegh Moshen Salehi (Bloomfield, NJ); Michal Sofka (Princeton, NJ); Prantik Kundu (Branford, CT)
Assignee: Hyperfine Operations, Inc.
G01R33/56G06N3/045G06N3/08G06T5/70G06T2207/10088G06T2207/20081G06T2207/20084
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,228,629
App. No.
17/496,104
Filed
Oct 7, 2021
Granted
Feb 18, 2025
Kind
B2
Art Unit
2663
USPC
382/100
Abstract

Techniques for denoising a magnetic resonance (MR) image are provided, including: obtaining a noisy MR image; denoising the noisy MR image of the subject using a denoising neural network model, and outputting a denoised MR image. The denoising neural network model is trained by: generating first training data for training a first neural network model to denoise MR images by generating a first plurality of noisy MR images using clean MR data associated with a source domain and first MR noise data associated with the target domain; training the first neural network model using the first training data; generating training data for training the denoising neural network model by applying the first neural network model to a second plurality of noisy MR images and generating a plurality of denoised MR images; and training the denoising neural network model using the training data for training the denoising neural network model.

Claims (76)

1. A method of denoising magnetic resonance (MR) images, the method comprising:

using at least one computer hardware processor to perform:

obtaining a noisy MR image of a subject, the noisy MR image associated with a target domain;

denoising the noisy MR image of the subject using a denoising neural network model to obtain a denoised MR image, the denoising neural network model trained by:

generating first training data for training a first neural network model to denoise MR images at least in part by generating a first plurality of noisy MR images using: (1) clean MR data associated with a source domain, wherein the clean MR data associated with the source domain comprises MR data that is collected using a magnetic resonance imaging (MRI) system having a main magnetic field strength of 0.5 T or greater; and (2) first MR noise data associated with the target domain;

training the first neural network model using the first training data;

generating training data for training the denoising neural network model at least in part by applying the first neural network model to a second plurality of noisy MR images and generating a corresponding plurality of denoised MR images, wherein the second plurality of noisy MR images is generated using second noisy MR data associated with the target domain, and wherein the second noisy MR data associated with the target domain comprises MR data that is collected using an MRI system having a main magnetic field strength greater than or equal to 20 mT and less than or equal to 0.2 T; and

training the denoising neural network model using the training data for training the denoising neural network model; and

outputting the denoised MR image.

2. The method of claim 1 , wherein the first training data comprises the first plurality of noisy MR images and a corresponding plurality of clean MR images, and wherein generating the first training data comprises:

generating first noisy MR data using the clean MR data associated with the source domain and the first MR noise data associated with the target domain;

generating the first plurality of noisy MR images by applying a reconstruction procedure to the first noisy MR data; and

generating the plurality of clean MR images by applying the reconstruction procedure to the clean MR data associated with the source domain.

3. The method of claim 1 , further comprising:

generating second training data for training a second neural network model to denoise MR images at least in part by generating a plurality of doubly noisy MR images using:

(1) second noisy MR data associated with the target domain; and

(2) second MR noise data associated with the target domain; and

training the second neural network model using the second training data.

4. The method of claim 3 , wherein the second training data comprises the plurality of doubly noisy MR images and the second plurality of noisy MR images, wherein generating the second training data comprises:

generating doubly noisy MR data using the second noisy MR data associated with the target domain and the second MR noise data associated with the target domain;

generating the plurality of doubly noisy MR images by applying a reconstruction procedure to the doubly noisy MR data; and

generating the second plurality of noisy MR images by applying the reconstruction procedure to the second noisy MR data associated with the target domain.

5. The method of claim 4 , wherein generating the training data for training the denoising neural network model further comprises applying the second neural network model to the second plurality of noisy MR images.

6. The method of claim 1 , wherein generating the training data for training the denoising neural network model further comprises:

generating a plurality of augmented denoised MR images by:

applying one or more transformations to images of the plurality of denoised MR images to generate a plurality of transformed MR images; and

combining the plurality of transformed MR images with the plurality of denoised MR images to generate the plurality of augmented denoised MR images; and

generating clean MR data associated with the target domain by applying a nonuniform transformation to images of the plurality of augmented denoised MR images.

7. The method of claim 6 , wherein the training data for training the denoising neural network model comprises a plurality of noisy MR training images and a plurality of clean MR training images, wherein generating the training data for training the denoising neural network model further comprises:

generating clean MR training data by combining the clean MR data associated with the source domain with the clean MR data associated with the target domain;

generating noisy MR training data using the clean MR training data and third MR noise data associated with the target domain;

generating the plurality of noisy MR training images by applying a reconstruction procedure to the noisy MR training data; and

generating the plurality of clean MR training images by applying the reconstruction procedure to the clean MR training data associated with the target domain.

8. The method of claim 1 , wherein obtaining the noisy MR image of the subject comprises:

collecting first noisy MR data by using a magnetic resonance imaging (MRI) system to image the subject; and

generating the noisy MR image of the subject using the collected first noisy MR data.

9. The method of claim 8 , wherein the first noisy MR data was previously collected using the MRI system, and wherein obtaining the noisy MR image of the subject comprises:

accessing the first noisy MR data; and

generating the noisy MR image using the accessed first noisy MR data.

10. The method of claim 8 , wherein the first noisy MR data is collected by the MRI system using a diffusion weighted imaging (DWI) pulse sequence.

11. The method of claim 10 , wherein the first MR noise data is generated by performing an empirical measurement of noise within the MRI system while the MRI system is operated using the DWI pulse sequence.

12. The method of claim 1 , wherein:

the clean MR data associated with the source domain comprises MR data that is collected by imaging a first portion of anatomy of the subject;

the second plurality of noisy MR images is generated using second noisy MR data associated with the target domain; and

the second noisy MR data associated with the target domain comprises MR data that is collected by imaging a second portion of anatomy different than the first portion of anatomy of the subject.

13. The method of claim 1 , wherein:

the clean MR data associated with the source domain comprises MR data that is collected using a first pulse sequence;

the second plurality of noisy MR images is generated using second noisy MR data associated with the target domain; and

the second noisy MR data associated with the target domain comprises MR data that is collected using a second pulse sequence different than the first pulse sequence.

14. At least one non-transitory computer readable storage medium storing processor-executable instructions that, when executed by at least one processor, cause the at least one processor to perform a method of denoising magnetic resonance (MR) images, the method comprising:

obtaining a noisy MR image of a subject, the noisy MR image associated with a target domain;

denoising the noisy MR image of the subject using a denoising neural network model to obtain a denoised MR image, the denoising neural network model trained by:

generating first training data for training a first neural network model to denoise MR images at least in part by generating a first plurality of noisy MR images using: (1) clean MR data associated with a source domain, wherein the clean MR data associated with the source domain comprises MR data that is collected using a magnetic resonance imaging (MRI) system having a main magnetic field strength of 0.5 T or greater; and (2) first MR noise data associated with the target domain;

training the first neural network model using the first training data;

generating training data for training the denoising neural network model at least in part by applying the first neural network model to a second plurality of noisy MR images and generating a corresponding plurality of denoised MR images, wherein the second plurality of noisy MR images is generated using second noisy MR data associated with the target domain, and wherein the second noisy MR data associated with the target domain comprises MR data that is collected using an MRI system having a main magnetic field strength greater than or equal to 20 mT and less than or equal to 0.2 T; and

training the denoising neural network model using the training data for training the denoising neural network model; and

outputting the denoised MR image.

15. The at least one non-transitory computer readable storage medium of claim 14 , wherein the first training data comprises the first plurality of noisy MR images and a corresponding plurality of clean MR images, and wherein generating the first training data comprises:

generating first noisy MR data using the clean MR data associated with the source domain and the first MR noise data associated with the target domain;

generating the first plurality of noisy MR images by applying a reconstruction procedure to the first noisy MR data; and

generating the plurality of clean MR images by applying the reconstruction procedure to the clean MR data associated with the source domain.

16. The at least one non-transitory computer readable storage medium of claim 14 , further comprising training the denoising neural network model by:

generating the first training data for training a first neural network model to denoise MR images at least in part by generating the first plurality of noisy MR images using: (1) the clean MR data associated with a source domain; and (2) the first MR noise data associated with a target domain;

training the first neural network model using the first training data;

generating the training data for training the denoising neural network model at least in part by applying the first neural network model to the second plurality of noisy MR images and generating the corresponding plurality of denoised MR images; and

training the denoising neural network model using the training data for training the denoising neural network model.

17. A magnetic resonance imaging (MRI) system, comprising:

a magnetics system having a plurality of magnetics components to produce magnetic fields for performing MRI; and

at least one processor configured to perform:

obtaining a noisy MR image of a subject, the noisy MR image associated with a target domain;

denoising the noisy MR image of the subject using a denoising neural network model to obtain a denoised MR image, the denoising neural network model trained by:

generating first training data for training a first neural network model to denoise MR images at least in part by generating a first plurality of noisy MR images using: (1) clean MR data associated with a source domain, wherein the clean MR data associated with the source domain comprises MR data that is collected using a magnetic resonance imaging (MRI) system having a main magnetic field strength of 0.5 T or greater; and (2) first MR noise data associated with the target domain;

training the first neural network model using the first training data;

generating training data for training the denoising neural network model at least in part by applying the first neural network model to a second plurality of noisy MR images and generating a corresponding plurality of denoised MR images, wherein the second plurality of noisy MR images is generated using second noisy MR data associated with the target domain, and wherein the second noisy MR data associated with the target domain comprises MR data that is collected using an MRI system having a main magnetic field strength greater than or equal to 20 mT and less than or equal to 0.2 T; and

training the denoising neural network model using the training data for training the denoising neural network model; and

outputting the denoised MR image.

Assignments (2)
CHANGE OF NAME Recorded Apr 18, 2022
From: HYPERFINE, INC.
To: HYPERFINE OPERATIONS, INC.
Reel/Frame 059724/0153 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 1, 2022
From: DEY, NEEL; SCHLEMPER, JO; MOHSENI SALEHI, SEYED SADEGH; SOFKA, MICHAL; KUNDU, PRANTIK
To: HYPERFINE, INC.
Reel/Frame 059469/0645 →
Continuity (3)
Provisional Application 63155696 · Mar 2, 2021
Provisional Application 63088672 · Oct 7, 2020
Related Publication 20220107378A1 · Apr 7, 2022
References Cited (58)
US 9625543B2 · Rearick et al. · 2017 [cited by applicant]
US 10222434B2 · Poole et al. · 2019 [cited by applicant]
US 10635943B1 · Lebel · 2020 [cited by examiner]
US 10726334B1 · Elnahrawy · 2020 [cited by examiner]
US 20050248347A1 · Damadian · 2005 [cited by applicant]
US 20150005618A1 · Dumoulin · 2015 [cited by applicant]
US 20150226817A1 · Pourrahimi · 2015 [cited by applicant]
US 20170213321A1 · Matviychuk et al. · 2017 [cited by applicant]
US 20190156524A1 · Park · 2019 [cited by examiner]
US 20200034998A1 · Schlemper et al. · 2020 [cited by applicant]
US 20200058106A1 · Lazarus et al. · 2020 [cited by applicant]
US 20200065940A1 · Tang · 2020 [cited by examiner]
US 20200289019A1 · Schlemper et al. · 2020 [cited by applicant]
US 20200294282A1 · Schlemper et al. · 2020 [cited by applicant]
US 20200311878A1 · Matsuura · 2020 [cited by examiner]
US 20200355761A1 · McNulty et al. · 2020 [cited by applicant]
US 20220188602A1 · Meyer · 2022 [cited by examiner]
US 20220354378A1 · Nacev et al. · 2022 [cited by applicant]
US 20240168105A1 · Inglis et al. · 2024 [cited by applicant]
WO WO2018210978A1 · 2018 [cited by applicant]
Invitation to Pay Additional Fees for International Application No. PCT/US2021/053918 dated Dec. 17, 2021. [cited by applicant]
International Search Report and Written Opinion for International Application No. PCT/US2021/053918 dated Feb. 8, 2022. [cited by applicant]
Batson et al., Noise2Self: Blind Denoising by Self-Supervision. arXiv preprint arXiv:1901.11365. Jun. 2019. 16 pages. [cited by applicant]
Dabov et al., Image denoising by sparse 3D transform-domain collaborative filtering. IEEE Transaction on Image Processing. Aug. 2007. 16(8). 16 pages. [cited by applicant]
Fish et al., Blind deconvolution by means of the Richardson-Lucy algorithm. Josa A. Jan. 1, 1995;12(1):58-65. [cited by applicant]
Gidel et al., Negative Momentum for Improved Game Dynamics. arXiv preprint arXiv:1807.04740. Aug. 2020. 26 pages. [cited by applicant]
Gisbert et al., Self-supervised Denoising via Diffeomorphic Template Estimation: Application to Optical Coherence Tomography. arXiv preprint arXiv:2008.08024. Aug. 18, 2020. 11 pages. [cited by applicant]
Glasser et al., The minimal preprocessing pipelines for the Human Connectome Project. Neuroimage. Oct. 15, 2013;80:105-24. [cited by applicant]
Goodfellow et al., Generative adversarial nets. Advances in neural information processing systems. 2014. 9 pages. [cited by applicant]
Gulrajani et al., Improved Training of Wasserstein GANs. arXiv preprint arXiv:1704.00028. Dec. 25, 2017. 20 pages. [cited by applicant]
Hendriksen et al., Noise2Inverse: Self-supervised deep convolutional denoising for tomography. arXiv preprint arXiv:2001.11801. Sep. 15, 2020. 16 pages. [cited by applicant]
Henry et al., Field-frequency locked in vivo proton MRS on a whole-body spectrometer. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine. Oct. 1999;42(4):… [cited by applicant]
Kingma et al., Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. Jan. 30, 2017. 15 pages. [cited by applicant]
Krizhevsky et al., Learning multiple layers of features from tiny images. MIT. 2009. 60 pages. [cited by applicant]
Krull et al., Noise2void-learning denoising from single noisy images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2019:2129-37. [cited by applicant]
Laine et al., High-Quality Self-Supervised Deep Image Denoising. arXiv preprint arXiv:1901.10277. Oct. 28, 2019. 16 pages. [cited by applicant]
LeCun et al., Gradient-based learning applied to document recognition. Proceedings of the IEEE. Nov. 1998;86(11):2278-324. [cited by applicant]
Lehtinen et al., Noise2Noise: Learning image restoration without clean data. arXiv preprint arXiv:1803.04189. Oct. 29, 2018. 12 pages. [cited by applicant]
Lustig et al., Compressed Sensing MRI. IEEE Signal Processing Magazine. Mar. 2008;72-82. [cited by applicant]
Mao et al., Least squares generative adversarial networks. Proceedings of the IEEE international conference on computer vision 2017;2794-2802. [cited by applicant]
McVeigh et al., Noise and filtration in magnetic resonance imaging. Medical physics. Sep. 1985;12(5):586-91. [cited by applicant]
Miyato et al., Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957. Feb. 16, 2018. 26 pages. [cited by applicant]
Mohan et al., Robust and interpretable blind image denoising via bias-free convolutional neural networks. arXiv preprint arXiv:1906.05478. Feb. 2020. 22 pages. [cited by applicant]
Moran et al., Noisier2noise: Learning to denoise from unpaired noisy data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020;12064-72. [cited by applicant]
Pruessmann et al., Advances in sensitivity encoding with arbitrary k-space trajectories. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine. Oct. 2001;46(… [cited by applicant]
Schaefer et al., Diffusion-weighted MR imaging of the brain. Radiology. Nov. 2000;217(2):331-45. [cited by applicant]
Schlemper et al., A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE transactions on Medical Imaging. Oct. 13, 2017;37(2):491-503. [cited by applicant]
Schlemper et al., Nonuniform variational network: deep learning for accelerated nonuniform MR image reconstruction. International Conference on Medical Image Computing and Computer-Assisted Intervention Oct. 13, 2019;57… [cited by applicant]
Schonfeld et al., A u-net based discriminator for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020;8204-13. [cited by applicant]
Walsh et al., Adaptive reconstruction of phased array MR imagery. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine. May 2000;43(5):682-90. [cited by applicant]
Wang et al., Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing. Apr. 13, 2004;13(4):600-12. [cited by applicant]
Wu et al., Group normalization. Proceedings of the European conference on computer vision (ECCV) 2018. 17 pages. [cited by applicant]
Zhang et al., Harmonic unpaired image-to-image translation. arXiv preprint arXiv:1902.09727. Feb. 26, 2019. 19 pages. [cited by applicant]
Zhang et al., Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing. Feb. 1, 2017;26(7):3142-55. [cited by applicant]
Zhu et al., Image reconstruction by domain-transform manifold learning. Nature. Mar. 2018;555(7697):487-92. [cited by applicant]
Zwart et al., Efficient sample density estimation by combining gridding and an optimized kernel. Magnetic resonance in medicine. Mar. 2012;67(3):701-10. [cited by applicant]
European extended Search Report on EP 21878515.2 DTD Oct. 10, 2024. [cited by applicant]
Manimala M. V. R. et al.: Sparse MR Image Reconstruction Considering Rician Noise Models : A Cnn Approach 11, Wireless Personal Communications., vol. 116, No. 1, Aug. 11, 2020 (Aug. 11, 2020), pp. 491-511. [cited by applicant]