IP Library › Granted Patent US 12,272,030
Granted Patent B2
US 12,272,030 · App. 17/445,382 · Granted Apr 8, 2025

Building units for machine learning models for denoising images and systems and methods for using same

Inventors: Bambi L DeLaRosa (Boise, ID); Katya Giannios (Boise, ID); Abhishek Chaurasia (Boise, ID)
Assignee: Micron Technology, Inc.
G06T5/70G06N3/048G06N3/08G06T5/50G06T7/0012G06T2207/10061G06T2207/20081G06T2207/20084G06T2207/20216
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,272,030
App. No.
17/445,382
Granted
Apr 8, 2025
Kind
B2
Abstract

In some examples, a machine learning model may be trained to denoise an image. In some examples, the machine learning model may identify noise in an image of a sequence based at least in part, on at least one other image of the sequence. In some examples, the machine learning model may include a recurrent neural network. In some examples, the machine learning model may have a modular architecture including one or more building units. In some examples, the machine learning model may have a multi-branch architecture. In some examples, the noise may be identified and removed from the image by an iterative process.

Claims (74)

1. An apparatus comprising:

a logic circuit configured to receive a first input and a second input comprising information related to an image and further configured to provide a first output, a second output, and a noise portion of the image;

a convolver configured to convolve the first input to generate the first output;

a synthesizer configured to receive the first output and the second input to generate the second output;

a noise attention block configured to provide the noise portion based, at least in part, on the second output; and

an adder configured to combine the noise portion and the image to produce an output image, wherein the output image has less noise than the image.

2. The apparatus of claim 1 , wherein the synthesizer comprises:

an update gate configured to determine an amount of the first input and an amount of the second input to include in the second output; and

a reset gate configured to determine an amount of the first input and an amount of the second input to exclude from the second output.

3. The apparatus of claim 2 , wherein the update gate and the reset gate each comprise:

a first convolution block configured to convolve the first output;

a second convolution block configured to convolve the second input; and

an arithmetic block configured to combine an output of the first convolution block and the second convolution block.

4. The apparatus of claim 3 , wherein the update gate further comprises an activation block configured to apply an activation function to an output of the arithmetic block.

5. The apparatus of claim 4 , wherein the activation function comprises logic configured to implement a sigmoid function.

6. The apparatus of claim 3 , wherein the arithmetic block comprises logic configured to implement a summing function.

7. The apparatus claim 2 , wherein the synthesizer further comprises a current cell configured to receive the first input, the second input, an output of the update gate, and an output of the reset gate,

wherein the second output is based, at least in part, on the first output, the output of the update gate, and an output of the current cell.

8. The apparatus of claim 7 , wherein the current cell comprises:

a first convolution block configured to convolve the first output;

a second convolution block configured to convolve the second input;

a first arithmetic block configured to combine an output of the first convolution block and the output of the reset gate; and

a second arithmetic block configured to combine an output of the second convolution block and an output of the first arithmetic block.

9. The apparatus of claim 8 , wherein the current cell further comprises an activation block configured to apply an activation function to an output of the second arithmetic block.

10. The apparatus of claim 9 , wherein the activation function comprises logic configured to implement a hyperbolic tangent function.

11. The apparatus of claim 8 , wherein the first arithmetic block comprises logic configured to implement an elementwise multiplication function.

12. The apparatus of claim 8 , wherein the second arithmetic block comprises logic configured to implement a summing function.

13. The apparatus of claim 7 , wherein the synthesizer further comprises:

a first arithmetic block configured to combine the first output and the output of the update gate;

a unity function block configured to apply a unity function to the output of the update gate;

a second arithmetic block configured to combine the output of the current cell and an output of the unity function block; and

a third arithmetic block configured to combine an output of the first arithmetic block and the second arithmetic block to provide the second output.

14. The apparatus of claim 13 , wherein the first arithmetic block comprises logic configured to implement an elementwise multiplication function.

15. The apparatus of claim 13 , wherein the second arithmetic block comprises logic configured to implement an elementwise multiplication function.

16. The apparatus of claim 13 , wherein the third arithmetic block comprises logic configured to implement a summing function.

17. The apparatus of claim 1 , wherein the convolver comprises:

a convolution block configured to convolve the first input; and

an activation block configured to activate an output of the convolution block.

18. The apparatus of claim 17 , wherein the activation block comprises logic configured to implement a hyperbolic tangent function.

19. The apparatus of claim 1 , wherein the noise attention block comprises:

a max pool block configured to max pool a plurality of channels of the second output;

an activation block configured to generate an attention map;

an extraction block configured to extract a first channel from the plurality of channels; and

an arithmetic block configured to elementwise multiply the first channel and the attention map to provide the noise portion.

20. The apparatus of claim 1 , further comprising a feedback block configured to combine the second output and an output of the adder block.

21. The apparatus of claim 20 , wherein the feedback block comprises:

an activation block configured to apply an activation function to the output of the adder block to generate an activated output; and

a copy block configured to copy over a first channel of the second output with the activated output.

22. A system comprising:

at least one processor; and

at least one non-transitory medium accessible to the processor, the at least one non-transitory medium encoded with instructions that, when executed, cause the system to implement a machine learning model, wherein the machine learning model comprises:

a plurality of building units, wherein individual ones of the building units are configured to receive a first input based on a second image and a second input based at least in part, on the image, and further configured to provide a first output, a second output, and a noise portion; and

a plurality of adders configured to combine the noise portions provided by the plurality of building units to the image to provide output images.

23. The system of claim 22 , wherein individual ones of the plurality of building units comprise a noise attention block configured to provide the noise portion based, at least in part, on the second output.

24. The system of claim 22 , wherein the machine learning model further comprises a plurality of feedback blocks configured to combine the second outputs provided by the plurality of building units with the output images of corresponding ones of the plurality of adder blocks.

25. The system of claim 24 , wherein outputs of at least some of the plurality of feedback blocks comprise the second inputs for at least some of the plurality of building units.

26. The system of claim 22 , wherein the second output is based, at least in part, on the first input, the second input, and the first output.

27. The system of claim 22 , wherein the first output is based, at least in part, on a convolution of the first input.

28. The system of claim 22 , wherein a number of the plurality of building units is based, at least in part, on a magnitude of noise included in the image.

29. The system of claim 22 , further comprising:

a second plurality of building units, wherein individual ones of the building units are configured to receive a third input based on a third image and a fourth input based at least in part, on the image, and further configured to provide a third output, a fourth output, and a second noise portion; and

a second plurality of adder blocks configured to combine the second noise portions provided by the second plurality of building units to the image to provide second output images.

30. The system of claim 29 , wherein the machine learning model is configured to combine a first output image of a final one of the plurality of adder blocks and a second output image of a final one of the second plurality of adder blocks to provide a final output image.

31. The system of claim 30 , wherein the final output image comprises an average of the first output image and the second output image.

32. The system of claim 22 , further comprising an ion beam scanning electron microscope configured to acquire the image and the second image.

33. The system of claim 22 , wherein the image and the second image are included in a sequence of images.

34. The system of claim 33 , wherein the sequence of images comprises a plurality of image planes acquired from a volume.

35. The system of claim 22 , wherein individual ones of the building units comprise:

an update gate configured to determine an amount of the first input and an amount of the second input to include in the second output;

a reset gate configured to determine an amount of the first input and an amount of the second input to exclude from the second output; and

a current cell configured to receive the first input, the second input, an output of the update gate, and an output of the reset gate,

wherein the second output is based, at least in part, on the first output, the output of the update gate, and an output of the current cell.

36. The system of claim 22 , wherein the instructions when executed, further cause the system to determine when a cancer cell is present in one or more of the output images.

37. The system of claim 22 , wherein the instructions when executed, further cause the system to provide at least one of a diagnosis or treatment recommendation based, at least in part, on the output images.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 18, 2021
From: DELAROSA, BAMBI L.; GIANNIOS, KATYA; CHAURASIA, ABHISHEK
To: MICRON TECHNOLOGY, INC.
Reel/Frame 057220/0001 →
Continuity (4)
Provisional Application 63163678 · Mar 19, 2021
Provisional Application 63163688 · Mar 19, 2021
Provisional Application 63163682 · Mar 19, 2021
Related Publication 20220309618A1 · Sep 29, 2022
References Cited (98)
US 5383457A · Cohen · 1995 [cited by examiner]
US 9792900B1 · Kaskari et al. · 2017 [cited by applicant]
US 10063255B2 · Riedel · 2018 [cited by examiner]
US 10726525B2 · El-Khamy et al. · 2020 [cited by applicant]
US 11330378B1 · Jelčicová et al. · 2022 [cited by examiner]
US 11657504B1 · Moinuddin · 2023 [cited by examiner]
US 11790492B1 · Ahmad et al. · 2023 [cited by applicant]
US 12086703B2 · Delarosa et al. · 2024 [cited by applicant]
US 20040071367A1 · Irani et al. · 2004 [cited by applicant]
US 20060285020A1 · Shin et al. · 2006 [cited by applicant]
US 20130266057A1 · Kokaram et al. · 2013 [cited by applicant]
US 20130282297A1 · Black et al. · 2013 [cited by applicant]
US 20170140897A1 · Phaneuf et al. · 2017 [cited by applicant]
US 20170237484A1 · Heath · 2017 [cited by examiner]
US 20170301342A1 · Kaskari et al. · 2017 [cited by applicant]
US 20170301344A1 · Kaskari · 2017 [cited by examiner]
US 20170359467A1 · Norris · 2017 [cited by examiner]
US 20180012460A1 · Heitz, III · 2018 [cited by examiner]
US 20180293713A1 · Vogels et al. · 2018 [cited by applicant]
US 20190086496A1 · Moeller et al. · 2019 [cited by applicant]
US 20190087646A1 · Goulden · 2019 [cited by examiner]
US 20190138887A1 · Salem · 2019 [cited by applicant]
US 20190228285A1 · Zhang · 2019 [cited by examiner]
US 20190304067A1 · Vogels et al. · 2019 [cited by applicant]
US 20190304069A1 · Vogels et al. · 2019 [cited by applicant]
US 20190333199A1 · Ozcan et al. · 2019 [cited by applicant]
US 20190365341A1 · Chan et al. · 2019 [cited by applicant]
US 20200005468A1 · Paul · 2020 [cited by examiner]
US 20200021808A1 · Mora · 2020 [cited by applicant]
US 20200065633A1 · Tiecke et al. · 2020 [cited by applicant]
US 20200267339A1 · Douady-Pleven et al. · 2020 [cited by applicant]
US 20200342748A1 · Tournier · 2020 [cited by examiner]
US 20200349675A1 · Park et al. · 2020 [cited by applicant]
US 20200357099A1 · Long et al. · 2020 [cited by applicant]
US 20200388011A1 · Nair et al. · 2020 [cited by applicant]
US 20210012463A1 · Ramani et al. · 2021 [cited by applicant]
US 20210035268A1 · Xu et al. · 2021 [cited by applicant]
US 20210042996A1 · Das et al. · 2021 [cited by applicant]
US 20210052226A1 · Jauss et al. · 2021 [cited by applicant]
US 20210150674A1 · Cai et al. · 2021 [cited by applicant]
US 20210158048A1 · Lee · 2021 [cited by examiner]
US 20210290191A1 · Qi et al. · 2021 [cited by applicant]
US 20210304364A1 · Kabaria et al. · 2021 [cited by applicant]
US 20220044362A1 · Sharoukhov et al. · 2022 [cited by applicant]
US 20220107378A1 · Dey et al. · 2022 [cited by applicant]
US 20220180572A1 · Maheshwari · 2022 [cited by examiner]
US 20220188608A1 · Heinrich · 2022 [cited by examiner]
US 20220189100A1 · Kosomaa et al. · 2022 [cited by applicant]
US 20220300789A1 · DeLaRosa · 2022 [cited by examiner]
US 20220300791A1 · DeLaRosa · 2022 [cited by examiner]
US 20220301112A1 · DeLaRosa · 2022 [cited by examiner]
US 20220301113A1 · Delarosa et al. · 2022 [cited by applicant]
US 20220301114A1 · Marras et al. · 2022 [cited by applicant]
US 20230005106A1 · Buurma · 2023 [cited by examiner]
US 20230033442A1 · Xiang · 2023 [cited by examiner]
US 20230034382A1 · Kurata · 2023 [cited by examiner]
US 20230196515A1 · Koch et al. · 2023 [cited by applicant]
US 20230385647A1 · Liu · 2023 [cited by examiner]
US 20240020796A1 · Marras et al. · 2024 [cited by applicant]
US 20240054615A1 · Young · 2024 [cited by examiner]
CN 102175625A · 2011 [cited by applicant]
CN 107492090A · 2017 [cited by applicant]
CN 109978778A · 2019 [cited by applicant]
CN 111598787A · 2020 [cited by applicant]
CN 111860406A · 2020 [cited by applicant]
WO 2020016337A1 · 2020 [cited by applicant]
WO 2022197476A1 · 2022 [cited by applicant]
WO 2022197477A1 · 2022 [cited by applicant]
WO 2022197478A1 · 2022 [cited by applicant]
Muhammad Sufian et al. “Denoising the Wireless Channel Corrupted Images Using Maching Learning” 2019 20th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Dist… [cited by applicant]
Shiwei Zhou et al. “Multi-View Image Denoising Using Convolutional Neural Network” Sensors; Jun. 2019; pp. all. [cited by applicant]
Unpublished PCT Application No. PCT/US2022/019124 filed Mar. 7, 2022; pp. all. [cited by applicant]
Unpublished PCT Application No. PCT/US2022/019130 filed Mar. 7, 2022; pp. all. [cited by applicant]
Unpublished PCT Application No. PCT/US2022/019136 filed Mar. 7, 2022; pp. all. [cited by applicant]
Hosseinkhani, Zohreh , et al., “Adaptive Real-Time Removal of Impulse Noise in Medical Images”, Retrieved from https://arxiv.org/abs/1709.02270, Oct. 2017, 9 pages. [cited by applicant]
Maji, Suman Kumar, et al., “A Feature Based Reconstruction Model for Fluorescence Microscopy Image Denoising”, Scientific Reports, vol. 9, Art. No. 7725, May 2019, 9 pages. [cited by applicant]
Roels, Joris , et al., “An Interactive IMAGEj Plugin for Semi-Automated Image Denoising in Electron Microscopy”, Nature Communications, vol. 11, Art. No. 771, Feb. 7, 2020, 13 pages. [cited by applicant]
George Cazenavette et al. “Architectural Adversarial Robustness: The Case for Deep Pursuit”; Nov. 2020; Cornell University Computer Science; pp. all. [cited by applicant]
Kenzo Isogawa et al. “Deep Shrinkage Convolutional Neural Network for Adaptive Noise Reduction”; IEEE Signal Processing Letters; vol. 25, No. 2; Feb. 2018; pp. all. [cited by applicant]
Sara Cavallaro et al. “High Throughput Imaging of Nanoscale Extracellular Vesticles by Scanning Electron Microscopy for Accurate Size-Based Profiling and Morphological Analysis”; Cold Spring Harbor Laboratory; Jan. 2021… [cited by applicant]
U.S. Appl. No. 17/445,376 titled “Modular Machine Learning Models for Denoising Images and Systems and Methods for Using Same”, filed Aug. 18, 2021. [cited by applicant]
U.S. Appl. No. 17/445,377 titled “Building Units for Machine Learning Models for Denoising Images and Systemsand Methods for Using Same”, filed Aug. 18, 2021. [cited by applicant]
U.S. Appl. No. 17/445,379 titled “Systems and Methods for Training Machine Learning Models for Denoisingimages”, filed Aug. 18, 2021. [cited by applicant]
U.S. Appl. No. 17/445,380 titled “Modular Machine Learning Models for Denoising Images and Systems Andmethods for Using Same”, filed Aug. 18, 2021. [cited by applicant]
Jain, Viren et al., “Natural Image Denoising with Convolutional Networks”, Advances in Neural Information Processing Systems, 2009, 769-776. [cited by applicant]
Krull, Alexander et al., “Noise2Void—Learning Denoising From Single Noisy Images”, MPI-CBG/PKS (CSBD), Dresden, Germany, arXiv e-prints, arXiv:1811.10980, Nov. 2018, 1-9. [cited by applicant]
Lehtinen, Jaakko et al., “Noise2Noise: Learning Image Restoration Without Clean Data”, https://github.com/NVlabs/noise2noise, arXiv e-prints, arXiv:1803.04189, Mar. 2018, 1-12. [cited by applicant]
Remez, Tal et al., “Class-Aware Fully-Convolutional Gaussian And Poisson Denoising”, IEEE Transactions on Image Processing, 27(11), Nov. 2018, 5707-5722. [cited by applicant]
Remez, Tal et al., “Deep Class Aware Denoising”, School of Electrical Engineering, Tel-Aviv University, Israel, Computer Science Department, Technion—IIT, Israel, arXiv e-prints, arXiv:1701.01698, Jan. 2017, 1-23. [cited by applicant]
Riesterer, Jessica et al., “A Workflow For Visualizing Human Cancer Biopsies Using Large-Format Electron Microscopy”, bioRxiv, 2009. [cited by applicant]
Ronneberger, Olaf et al., “U-Net: Convolutional Networks For Biomedical Image Segmentation”, Computer Science Department and BIOSS Centre for Biological Signalling Studies, University of Freiburg. Germany, arXiv e-print… [cited by applicant]
Shi, Xingjian et al., “Convolutional LSTM Network: A Machine Learning Approach For Precipitation Nowcasting”, Department of Computer Science and Engineering Hong Kong University of Science and Technology, rXiv:1506.0421… [cited by applicant]
Wu, Dufan et al., “Consensus Neural Network for Medical Imaging Denoising With Only Noisy Training Samples”, Center for Advanced Medical Computing and Analysis, Massachusetts General Hospital and Harvard Medical School,… [cited by applicant]
Yang, Dong et al., “Deep Image-to-Image Recurrent Network With Shape Basis Learning For Automatic Vertebra Labeling In Large-Scale 3D Ct Volumes”, Sep. 2017, 498-506. [cited by applicant]
Cazenavette, George , et al., “Architectural Adversarial Robustness: The Case for Deep Pursuit”, https://arxiv.org/abs/2011.14427, Nov. 2020, 11 pages. [cited by applicant]
Isogawa, Kenzo , et al., “Deep Shrinkage Convolutional Neural Network for Adaptive Noise Reduction”, IEEE Signal Processing Letters, vol. 25, No. 2, Feb. 2018, 5 pages. [cited by applicant]
Sufian, Muhammad , et al., “Denoising The Wireless Channel Corrupted Images Using Machine Learning”, 20th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Dist… [cited by applicant]
Cao, Chunshui , et al., “Feedback Convolutional Neural Network for Visual Localization and Segmentation”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, No. 7, Jul. 2019, 14 pages. [cited by applicant]
Cited By (1)
US 12,657,672