IP Library › Granted Patent US 12,277,683
Granted Patent B2
US 12,277,683 · App. 17/445,376 · Granted Apr 15, 2025

Modular machine learning models for denoising images and systems and methods for using same

Inventors: Bambi L DeLaRosa (Boise, ID); Katya Giannios (Boise, ID); Abhishek Chaurasia (Boise, ID)
Assignee: Micron Technology, Inc.
G06T5/70G06N3/04G06T5/50G06V20/695
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,277,683
App. No.
17/445,376
Granted
Apr 15, 2025
Kind
B2
Abstract

A machine learning model may be trained to denoise an image. The machine learning model may identify noise in an image of a sequence based at least in part, on at least one other image of the sequence. The machine learning model may include a recurrent neural network. The machine learning model may have a modular architecture including one or more building units. The machine learning model may have a multi-branch architecture. The noise may be identified and removed from the image by an iterative process.

Claims (69)

1. A system comprising:

at least one processor; and

at least one non-transitory medium accessible to the processor, the at least one non-transitory medium encoded with instructions that, when executed, cause the system to implement a machine learning model, wherein the machine learning model is configured to identify at least some noise of an image of a sequence of images based, at least in part, on the image and at least one other image of the sequence of images, wherein the machine learning model comprises;

a convolutional layer configured to convolve and linearly rectify the image and the at least one other image of the sequence of images to generate a first output;

a plurality of layers, wherein individual ones of the plurality of layers are configured to receive a memory output of a previous layer of the plurality of layers and provide, as an output, a portion of noise of the image, wherein a first layer of the plurality of layers is configured to receive the first output from the convolutional layer and wherein the machine learning model is configured to remove from the image the portion of noise provided by the previous layer of the plurality of layers to generate an intermediate image at each layer of the plurality of layers and provide the intermediate image to each next layer of the plurality of layers; and

a second convolutional layer configured to convolve an output of a last one of the plurality of layers and output a final portion of the noise of the image, wherein the machine learning model is configured to remove the final portion of the noise from the image.

2. The system of claim 1 , wherein the machine learning model is further configured to remove at least some of the noise from the image and provide an output image having reduced noise.

3. The system of claim 1 , wherein the sequence of images comprises a plurality of image planes acquired from a volume.

4. The system of claim 3 , wherein the plurality of image planes are spatially adjacent in the volume.

5. The system of claim 3 , wherein the image planes are spatially distinct.

6. The system of claim 1 , wherein the sequence of images comprises a plurality of temporally spaced images.

7. The system of claim 1 , wherein the machine learning model is configured to remove the portion of the noise received from the plurality of layers from the image and provide an output image having reduced noise.

8. The system of claim 1 , wherein the machine learning model further comprises a plurality of feedback blocks, wherein individual ones of the feedback blocks are configured to receive the memory output of the previous layer of the plurality of layers and the image with the portion of the noise provided by the previous layer of the plurality of layers removed and provide a feedback output to a current layer of the plurality of layers.

9. The system of claim 8 , wherein the second convolutional layer is configured to convolve the feedback output of a last one of the plurality of feedback blocks and output the final portion of the noise of the image.

10. The system of claim 1 , wherein a number of the plurality of layers is based, at least in part, on an amount of noise in the image.

11. The system of claim 1 , wherein the machine learning model is configured to remove the noise from the image of the sequence of images based on two other images of the sequence of images.

12. The system of claim 11 , wherein a first image of the two other images directly proceeds the image in the sequence of images and a second image of the two other images directly follows the image in the sequence of images.

13. A system comprising:

at least one processor; and

at least one non-transitory medium accessible to the processor, the at least one non-transitory medium encoded with instructions that, when executed, cause the system to implement a machine learning model, wherein the machine learning model is configured to identify at least some noise of an image of a sequence of images based, at least in part, on the image and at least one other image of the sequence of images, wherein the machine learning model comprises:

a first branch configured to remove the noise from the image based, at least in part, on the first image to provide a first intermediate image; and

a second branch configured to remove the noise from the image based, at least in part, on the second image to provide a second intermediate image,

wherein an output image comprises a combination of the first intermediate image and the second intermediate image and wherein the first branch and the second branch each comprise:

a convolutional layer configured to convolve and linearly rectify the image and the first image or the second image to generate a first output;

a plurality of layers, wherein individual ones of the plurality of layers are configured to receive a memory output of a previous layer of the plurality of layers and provide, as an output, a portion of noise of the image, wherein a first layer of the plurality of layers is configured to receive the first output from the convolutional layer and wherein the machine learning model is configured to remove from the image the portion of noise provided by the previous layer of the plurality of layers to generate an intermediate image at each layer of the plurality of layers and provide the intermediate image to each next layer of the plurality of layers; and

a second convolutional layer configured to convolve an output of a last one of the plurality of lavers and output a final portion of the noise of the image, wherein the machine learning model is configured to remove the final portion of the noise from the image.

14. A non-transitory computer readable medium encoded with instructions that when executed implement a machine learning model, the machine learning model comprising:

a convolutional layer configured to convolve and linearly rectify an image from a sequence of images and at least one other image of the sequence of images to generate a first output;

a plurality of layers, wherein individual ones of the plurality of layers are configured to receive a memory output of a previous layer of the plurality of layers, identify a portion of noise in the image, and provide, as an output, the portion of noise of the image, wherein a first layer of the plurality of layers is configured to receive the first output from the convolutional layer and wherein the machine learning model is configured to remove from the image the portion of noise provided by the previous layer of the plurality of layers to generate an intermediate image at each layer of the plurality of layers and provide the intermediate image to each next layer of the plurality of layers;

at least one arithmetic block configured to remove the portion of the noise received from the plurality of layers from the image to provide an output image; and

a second convolutional layer configured to convolve an output of a last one of the plurality of layers and output a final portion of the noise of the image, wherein the machine learning model is configured to remove the final portion of the noise from the image.

15. The non-transitory computer readable medium of claim 14 , the machine learning model further comprises a plurality of feedback blocks, wherein individual ones of the feedback blocks are configured to receive the memory output of the previous layer of the plurality of layers and the image with the portion of the noise provided by the previous layer of the plurality of layers removed and provide a feedback output to a current layer of the plurality of layers.

16. The non-transitory computer readable medium of claim 15 , the second convolutional layer configured to convolve the feedback output of a last one of the plurality of feedback blocks and output the final portion of the noise of the image, wherein the at least one arithmetic block is configured to remove the final portion of the noise from the image.

17. The non-transitory computer readable medium of claim 14 , wherein the at least one other image of the sequence of images comprises a first image and a second image, and wherein the machine learning model comprises:

a first branch configured to remove the noise from the image based, at least in part, on the first image to provide a first intermediate image; and

a second branch configured to remove the noise from the image based, at least in part, on the second image to provide a second intermediate image,

wherein the output image comprises a combination of the first intermediate image and the second intermediate image.

18. The non-transitory computer readable medium of claim 17 , wherein the first image directly proceeds the image in the sequence of images and the second image directly follows the image in the sequence of images.

19. The non-transitory computer readable medium of claim 14 , wherein individual ones of the plurality of layers comprises an update gate configured to determine an amount of the memory output received from the previous layer to retain in a memory output of a current layer of the plurality of layers.

20. The non-transitory computer readable medium of claim 14 , wherein individual ones of the plurality of layers comprises a reset gate configured to determine an amount of the memory output received from the previous layer to omit from a memory output of a current layer of the plurality of layers.

21. The non-transitory computer readable medium of claim 14 , wherein the noise is generated by a plurality of sources.

22. The non-transitory computer readable medium model of claim 14 , wherein the noise is random.

23. The non-transitory computer readable medium of claim 14 , wherein the machine learning model is trained by minimizing a loss function comprising an L2 norm weight decay regularizer.

24. A method comprising:

identifying noise in an image of a sequence of images based, at least in part, on the image and at least one other image of the sequence of images, wherein the image and the at least one other image are spatially distinct, temporally distinct, or a combination thereof; and

removing at least a portion of the noise from the image to provide an output image, wherein the identifying and the removing comprise:

convolving and linearly rectifying the image and the at least one other image of the sequence of images to generate a first output;

receiving, at a plurality of layers, a memory output of a previous layer of the plurality of layers and providing, as an output, a portion of noise of the image, wherein a first layer of the plurality of layers is configured to receive the first output and wherein removing the at least a portion of the noise from the image further comprises:

generating an intermediate image at each layer of the plurality of layers; and

providing the intermediate image to each next layer of the plurality of layers;

providing a feedback output to a current layer of the plurality of layers based, at least in part, the memory output of the previous layer of the plurality of layers and the intermediate image with the portion of the noise provided by the previous layer of the plurality of layers removed; and

removing the portion of the noise received from the plurality of layers from the image to provide the output image.

25. The method of claim 24 , wherein the identifying and removing are performed by a machine learning model implemented by at least one processor.

26. The method of claim 25 , wherein the machine learning model comprises a recurrent neural network.

27. The method of claim 24 ,

wherein the at least one other image of the sequence of images comprises a first image and a second image and the identifying and removing further comprise:

removing the noise from the image based, at least in part, on the first image to provide a first intermediate image;

removing the noise from the image based, at least in part, on the second image to provide a second intermediate image; and

combining the first intermediate image and the second intermediate image to provide the output image.

28. The method of claim 24 , further comprising acquiring the sequence of images with a medical imaging system.

29. The method of claim 28 , wherein the medical imaging system comprises a focused ion beam scanning electron microscope.

30. The method of claim 24 , wherein the sequence of images comprises a plurality of image planes acquired from a volume of a biological sample.

31. The method of claim 24 , further comprising:

subdividing the image into a first plurality of sub-images;

subdividing the at least one other image into a second plurality of sub-images;

performing the identifying and the removing with the first plurality of sub-images and the second plurality of sub-images to generate a plurality of output sub-images; and

reconstructing the plurality of output sub-images to provide the output image.

32. The method of claim 24 , further comprising detecting a cancer cell based, at least in part, on the output image.

33. The method of claim 24 , further comprising detecting a cell-to-cell interaction based, at least in part, on the output image.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 18, 2021
From: DELAROSA, BAMBI L.; GIANNIOS, KATYA; CHAURASIA, ABHISHEK
To: MICRON TECHNOLOGY, INC.
Reel/Frame 057219/0648 →
Continuity (4)
Provisional Application 63163688 · Mar 19, 2021
Provisional Application 63163678 · Mar 19, 2021
Provisional Application 63163682 · Mar 19, 2021
Related Publication 20220301112A1 · Sep 22, 2022
References Cited (100)
US 5383457A · Cohen · 1995 [cited by applicant]
US 9792900B1 · Kaskari et al. · 2017 [cited by applicant]
US 10063255B2 · Riedel et al. · 2018 [cited by applicant]
US 10726525B2 · El-Khamy et al. · 2020 [cited by applicant]
US 11330378B1 · Jelcicova et al. · 2022 [cited by applicant]
US 11657504B1 · Moinuddin et al. · 2023 [cited by applicant]
US 11790492B1 · Ahmad et al. · 2023 [cited by applicant]
US 12086703B2 · Delarosa et al. · 2024 [cited by applicant]
US 12148125B2 · Delarosa et al. · 2024 [cited by applicant]
US 20040071367A1 · Irani et al. · 2004 [cited by applicant]
US 20060285020A1 · Shin · 2006 [cited by examiner]
US 20130266057A1 · Kokaram et al. · 2013 [cited by applicant]
US 20130282297A1 · Black et al. · 2013 [cited by applicant]
US 20170140897A1 · Phaneuf · 2017 [cited by examiner]
US 20170237484A1 · Heath et al. · 2017 [cited by applicant]
US 20170301342A1 · Kaskari et al. · 2017 [cited by applicant]
US 20170301344A1 · Kaskari et al. · 2017 [cited by applicant]
US 20170359467A1 · Norris et al. · 2017 [cited by applicant]
US 20180012460A1 · Heitz et al. · 2018 [cited by applicant]
US 20180293713A1 · Vogels et al. · 2018 [cited by applicant]
US 20190086496A1 · Moeller et al. · 2019 [cited by applicant]
US 20190087646A1 · Goulden et al. · 2019 [cited by applicant]
US 20190138887A1 · Salem · 2019 [cited by examiner]
US 20190228285A1 · Zhang et al. · 2019 [cited by applicant]
US 20190304067A1 · Vogels et al. · 2019 [cited by applicant]
US 20190304069A1 · Vogels et al. · 2019 [cited by applicant]
US 20190333199A1 · Ozcan et al. · 2019 [cited by applicant]
US 20190365341A1 · Chan et al. · 2019 [cited by applicant]
US 20200005468A1 · Paul et al. · 2020 [cited by applicant]
US 20200021808A1 · Mora · 2020 [cited by applicant]
US 20200065633A1 · Tiecke · 2020 [cited by examiner]
US 20200267339A1 · Douady-Pleven et al. · 2020 [cited by applicant]
US 20200342748A1 · Tournier et al. · 2020 [cited by applicant]
US 20200349675A1 · Park et al. · 2020 [cited by applicant]
US 20200357099A1 · Long · 2020 [cited by examiner]
US 20200388011A1 · Nair et al. · 2020 [cited by applicant]
US 20210012463A1 · Ramani et al. · 2021 [cited by applicant]
US 20210035268A1 · Xu · 2021 [cited by examiner]
US 20210042996A1 · Das et al. · 2021 [cited by applicant]
US 20210052226A1 · Jauss · 2021 [cited by examiner]
US 20210150674A1 · Cai et al. · 2021 [cited by applicant]
US 20210158048A1 · Lee et al. · 2021 [cited by applicant]
US 20210290191A1 · Qi et al. · 2021 [cited by applicant]
US 20210304364A1 · Kabaria et al. · 2021 [cited by applicant]
US 20220030112A1 · Benkreira et al. · 2022 [cited by applicant]
US 20220044362A1 · Sharoukhov et al. · 2022 [cited by applicant]
US 20220107378A1 · Dey et al. · 2022 [cited by applicant]
US 20220180572A1 · Maheshwari et al. · 2022 [cited by applicant]
US 20220188608A1 · Heinrich et al. · 2022 [cited by applicant]
US 20220189100A1 · Kosomaa et al. · 2022 [cited by applicant]
US 20220300789A1 · Delarosa et al. · 2022 [cited by applicant]
US 20220300791A1 · Delarosa et al. · 2022 [cited by applicant]
US 20220301113A1 · DeLaRosa · 2022 [cited by examiner]
US 20220301114A1 · Marras et al. · 2022 [cited by applicant]
US 20220309618A1 · Delarosa et al. · 2022 [cited by applicant]
US 20230005106A1 · Buurma et al. · 2023 [cited by applicant]
US 20230033442A1 · Xiang et al. · 2023 [cited by applicant]
US 20230034382A1 · Kurata et al. · 2023 [cited by applicant]
US 20230196515A1 · Koch et al. · 2023 [cited by applicant]
US 20230385647A1 · Liu et al. · 2023 [cited by applicant]
US 20240020796A1 · Marras et al. · 2024 [cited by applicant]
US 20240054615A1 · Young et al. · 2024 [cited by applicant]
CN 102175625A · 2011 [cited by examiner]
CN 107492090A · 2017 [cited by applicant]
CN 109978778A · 2019 [cited by applicant]
CN 111598787A · 2020 [cited by applicant]
CN 111860406A · 2020 [cited by applicant]
WO WO2020016337A1 · 2020 [cited by examiner]
WO 2022197476A1 · 2022 [cited by applicant]
WO 2022197477A1 · 2022 [cited by applicant]
WO 2022197478A1 · 2022 [cited by applicant]
“George Cazenavette et. al., Architectural Adversarial Robustness: The Case for Deep Pursuit, Nov. 2020, arXiv: 2011. 14427, Cornell Univ. Computer Science” (Year: 2011). [cited by examiner]
“Kenzo Isogawa et. al., Deep Shrinkage Convolutional Neural Network for Adaptive Noise Reduction, IEEE Signal Processing Letters, vol. 25, No. 2, Feb. 2018” (Year: 2018). [cited by examiner]
“Sara Cavallaro et. al., High throughput imaging of nanoscale extracellular vesicles by scanning electron microscopy for accurate size-based profiling and morphological analysis, Cold Spring Harbor Laboratory, bioRxiv t… [cited by examiner]
“Muhammad Sufian et. al., Denoising The Wireless Channel Corrupted Images Using Machine Learning, 2019 20th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Di… [cited by examiner]
“Shiwei Zhou et. al., Multi-View Image Denoising Using Convolutional Neural Network, Jun. 2019, Sensors 2019, 19(11), 2597” (Year: 2019). [cited by examiner]
“Chunshui Cao et. al., Feedback Convolutional Neural Network for Visual Localization and Segmentation, Jul. 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, Issue 7” (Year: 2019). [cited by examiner]
International Search Report/Written Opinion dated Jun. 14, 2022 for PCT Application No. PCT/US2022/019124; pp. all. [cited by applicant]
Unpublished PCT Application No. PCT/US2022/019130 filed Mar. 7, 2022; pp. all. [cited by applicant]
Unpublished PCT Application No. PCT/US2022/019136 filed Mar. 7, 2022; pp. all. [cited by applicant]
Hosseinkhani, Zohreh , et al., “Adaptive Real-Time Removal of Impulse Noise in Medical Images”, Retrieved from https://arxiv.org/abs/1709.02270, Oct. 2017, 9 pages. [cited by applicant]
Maji, Suman Kumar, et al., “A Feature Based Reconstruction Model for Fluorescence Microscopy Image Denoising”, Scientific Reports, vol. 9, Art. No. 7725, May 2019, 9 pages. [cited by applicant]
Roels, Joris , et al., “An Interactive IMAGEj Plugin for Semi-Automated Image Denoising in Electron Microscopy”, Nature Communications, vol. 11, Art. No. 771, Feb. 7, 2020, 13 pages. [cited by applicant]
U.S. Appl. No. 17/445,377 titled “Building Units for Machine Learning Models for Denoising Images and Systemsand Methods for Using Same”, filed Aug. 18, 2021. [cited by applicant]
U.S. Appl. No. 17/445,379 titled “Systems and Methods for Training Machine Learning Models for Denoisingimages”, filed Aug. 18, 2021. [cited by applicant]
U.S. Appl. No. 17/445,380 titled “Modular Machine Learning Models for Denoising Images and Systems Andmethods for Using Same”, filed Aug. 18, 2021. [cited by applicant]
U.S. Appl. No. 17/445,382 titled “Building Units for Machine Learning Models for Denoising Images and Systemsand Methods for Using Same”, filed Aug. 18, 2021. [cited by applicant]
Jain, Viren et al., “Natural Image Denoising with Convolutional Networks”, Advances in Neural Information Processing Systems, 2009, 769-776. [cited by applicant]
Krull, Alexander et al., “Noise2Void—Learning Denoising From Single Noisy Images”, MPI-CBG/PKS (CSBD), Dresden, Germany, arXiv e-prints, arXiv:1811.10980, Nov. 2018, 1-9. [cited by applicant]
Lehtinen, Jaakko et al., “Noise2Noise: Learning Image Restoration Without Clean Data”, https://github.com/NVlabs/noise2noise, arXiv e-prints, arXiv:1803.04189, Mar. 2018, 1-12. [cited by applicant]
Remez, Tal et al., “Class-Aware Fully-Convolutional Gaussian And Poisson Denoising”, IEEE Transactions on Image Processing, 27(11), Nov. 2018, 5707-5722. [cited by applicant]
Remez, Tal et al., “Deep Class Aware Denoising”, School of Electrical Engineering, Tel-Aviv University, Israel, Computer Science Department, Technion—IIT, Israel, arXiv e-prints, arXiv:1701.01698, Jan. 2017, 1-23. [cited by applicant]
Riesterer, Jessica et al., “A Workflow For Visualizing Human Cancer Biopsies Using Large-Format Electron Microscopy”, bioRxiv, 2009. [cited by applicant]
Ronneberger, Olaf et al., “U-Net: Convolutional Networks For Biomedical Image Segmentation”, Computer Science Department and BIOSS Centre for Biological Signalling Studies, University of Freiburg. Germany, arXiv e-print… [cited by applicant]
Shi, Xingjian et al., “Convolutional LSTM Network: A Machine Learning Approach For Precipitation Nowcasting”, Department of Computer Science and Engineering Hong Kong University of Science and Technology, rXiv:1506.0421… [cited by applicant]
Wu, Dufan et al., “Consensus Neural Network for Medical Imaging Denoising With Only Noisy Training Samples”, Center for Advanced Medical Computing and Analysis, Massachusetts General Hospital and Harvard Medical School,… [cited by applicant]
Yang, Dong et al., “Deep Image-to-Image Recurrent Network With Shape Basis Learning For Automatic Vertebra Labeling In Large-Scale 3D Ct Volumes”, Sep. 2017, 498-506. [cited by applicant]
Cazenavette, George , et al., “Architectural Adversarial Robustness: The Case for Deep Pursuit”, https://arxiv.org/abs/2011.14427, Nov. 2020, 11 pages. [cited by applicant]
Isogawa, Kenzo , et al., “Deep Shrinkage Convolutional Neural Network for Adaptive Noise Reduction”, IEEE Signal Processing Letters, vol. 25, No. 2, Feb. 2018, 5 pages. [cited by applicant]
Sufian, Muhammad , et al., “Denoising The Wireless Channel Corrupted Images Using Machine Learning”, 20th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Dist… [cited by applicant]