Systems and methods for training machine learning models for denoising images
In some examples, a machine learning model may be trained to denoise an image. In some examples, the machine learning model may identify noise in an image of a sequence based at least in part, on at least one other image of the sequence. In some examples, the machine learning model may include a recurrent neural network. In some examples, the machine learning model may have a modular architecture including one or more building units. In some examples, the machine learning model may have a multi-branch architecture. In some examples, the noise may be identified and removed from the image by an iterative process.
1. A method comprising:
writing to a memory one or more bits of data that represent an initial value setting for a weight of a feature matrix of a machine learning model;
providing a first signal representative of a first image and a second image of an image sequence as inputs to circuitry configured as a first branch of the machine learning model;
providing a second signal representative of the first image and a third image of the image sequence as inputs to circuitry configured as a second branch of the machine learning model, wherein the first and second signals each comprise noise;
receiving a first output from the circuitry configured as the first branch and a second output from the circuitry configured as the second branch; and
writing to the memory one or more bits of data that represent an adjusted value of the weight different from the initial value, the adjusted value based, at least in part, on a characteristic of first output and the second output satisfying a threshold value, wherein the threshold value represents a minimum value of a loss fiction, wherein the loss function comprises a noise-to-noise term and a regularizer term.
2. The method of claim 1 , wherein the first image and the second image are consecutive images of the image sequence and the first image and the third image are consecutive images of the image sequence.
3. The method of claim 1 , wherein the noise-to-noise term is based, at least in part, on:
a first term comprising a difference of the second image and the second output;
a second term comprising a difference of the third image and the first output; and
a third term comprising a difference of the first output and the second output.
4. The method of claim 3 , wherein the noise-to-noise term comprises a function comprising a weighted sum of the first term, the second term, and the third term.
5. The method of claim 4 , wherein a weight of the first term and a weight of the second term are equal.
6. The method of claim 4 , wherein a weight of the third term is negative.
7. The method of claim 4 , wherein the noise-to-noise term comprises an average of the function.
8. The method of claim 1 , wherein the regularizer term comprises an L2 norm weight decay term.
9. The method of claim 1 , wherein the method is performed at least in part by an ADAM optimizer.
10. A non-transitory computer-readable medium encoded with instructions, wherein when the instructions are executed by at least one processor included in a computing system, cause the computing system to:
write in a memory an initial value of a weight of a feature matrix of a machine learning model;
provide a training data set to the memory as an input to the machine learning model, wherein the training data set comprises a plurality of image sequences, wherein images of the image sequence include noise;
receive a first output and a second output for individual ones of the plurality of image sequences from the machine learning model based at least in part on the training data set;
calculate a value of a loss function based, at least in part, on the first output and the second output; and
write an adjusted value of the weight to the memory based, at least in part, on the value of the loss function, wherein the loss function comprises a noise-to-noise term comprising:
a first term based at least in part on a first portion of the individual ones of the plurality of sequences and the second image and the second output:
a second term based at least in part on a second portion of the individual ones of the plurality of sequences and the first output; and
a third term based at least in part on a difference of the first output and the second output.
11. The non-transitory computer readable medium of claim 10 , wherein a first portion of individual ones of the plurality of image sequences are provided to a first branch of the machine learning model and a second portion of individual ones of the plurality of image sequences are provided to a second branch of the machine learning model,
wherein the first output is based, at least in part, on the first portion and the second output is based, at least in part, on the second portion.
12. The non-transitory computer readable medium of claim 11 , wherein the first portion and the second portion overlap by at least one image.
13. The non-transitory computer readable medium of claim 11 , wherein the first branch and the second branch of the machine learning model comprise a same architecture.
14. The non-transitory computer readable medium of claim 13 , wherein the architecture comprises a recurrent neural network.
15. The non-transitory computer readable medium of claim 10 , wherein the loss function further comprises a regularizer term summed with the noise-to-noise term.
16. The non-transitory computer readable medium of claim 15 , wherein the regularizer term comprises a weighted decay L2 norm.
17. The non-transitory computer readable medium of claim 10 , wherein the first output is an image of an individual one of the plurality of image sequences with at least a first portion of the noise removed and the second output is the image of the individual one of the plurality of image sequences with at least a second portion of the noise removed.
18. The non-transitory computer readable medium of claim 17 , wherein the first portion of the noise and the second portion of the noise are different.
19. A system comprising:
at least one non-transitory medium encoded with instructions;
at least one processor in communication with the non-transitory medium configured to execute the instructions, wherein when executed, the instructions cause the processor to:
while a value of a loss function stored in a memory is above a minimum value:
write to the memory an initial value of a weight of a feature matrix of the machine learning model;
provide a first plurality of images and a second plurality of images of a plurality of image sequences as a first plurality of inputs to the machine learning model;
provide the first plurality of images and a third plurality of images of the plurality of image sequences as a second plurality of inputs to the machine learning model, wherein the first, second, and third plurality of images include noise;
receive a first plurality of outputs responsive to the first plurality of inputs and a second plurality of outputs responsive to the second plurality of inputs;
calculate the value of the loss function based, at least in part, on the first plurality of outputs and the second plurality of outputs; and
write an adjusted value of the weight to the memory based, at least in part, on the value of the loss function,
wherein the loss function comprises a noise-to-noise term comprising;
1
N
∑
k
=
1
N
L
(
x
k
,
i
+
1
,
x
k
,
i
-
1
,
out
k
,
i
+
1
,
out
k
,
i
-
1
)
,
wherein xk,i+1 comprises the second plurality of images, xk,i−1 comprises the third plurality of images, outk,i+1 comprises the first plurality of outputs, and outk,i−1 comprises the second plurality of outputs.
20. The system of claim 19 , wherein the first plurality of inputs are provided to a first branch of the machine learning model and the second plurality of inputs are provided to a second branch of the machine learning model.
21. The system of claim 19 , wherein N is equal to a number of the plurality of image sequences.
22. The system of claim 19 wherein L(X k,i+1 , x k,i−1 , out k,i+1 , out k,i−1 ) comprises:
1
2
out
k
,
i
-
1
-
x
k
,
i
+
1
2
2
+
1
2
out
k
,
i
+
1
-
x
k
,
i
-
1
2
2
-
1
4
out
k
,
i
-
1
-
out
k
,
i
+
1
2
2
.
23. The system of claim 19 , wherein the loss function comprises a sum of the noise-to-noise term and a regularizer term.
24. The system of claim 19 , wherein while the value of the loss function is above the minimum value, the instructions further cause the processor to:
while the value of a loss function is above the minimum value:
set a plurality of weights of a plurality of features of a feature matrix of the machine learning model to a corresponding plurality of initial values; and
adjust the plurality of weights based, at least in part, on the value of the loss function.
25. The system of claim 19 , wherein while the value of the loss function is above the minimum value, the instructions further cause the processor to:
while the value of a loss function is above the minimum value:
set a plurality of weights of a corresponding plurality of feature matrices of the machine learning model to a corresponding plurality of initial values; and
adjust the plurality of weights based, at least in part, on the value of the loss function.
26. The system of claim 19 , wherein the first plurality of outputs comprise the first plurality of images with a first portion of the noise removed and the second plurality of outputs comprise the first plurality of images with a second portion of the noise removed.
27. The system of claim 19 , wherein when the value of the loss function is the minimum value, the instructions further cause the at least one processor to:
implement the machine learning model to identify and remove noise from a new image of a new sequence of images based, at least in part, on the new image, at least one other image of the new sequence of images, and the weight of the feature of the feature matrix to provide an output image.
28. The system of claim 27 , wherein the at least one other image of the new sequence of images comprises a first image and a second image.
29. The system of claim 28 , wherein the new image and the first image are provided as a first input to the machine learning model and the new image and the second image are provided as a second input to the machine learning model, wherein the machine learning model provides a first output based on the first input and a second output based on the second input,
wherein the output image is an average of the first output and the second output.
30. The system of claim 19 , wherein individual ones of the plurality of sequences of images comprise a plurality of image planes acquired from a volume.
31. The system of claim 19 , wherein individual ones of the plurality of sequences of images comprise a plurality of temporally spaced images.
32. The system of claim 19 , wherein the plurality of sequences of images comprise focused ion beam scanning electron microscopy images.
33. The system of claim 19 , further comprising a medical imaging system configured to acquire the plurality of sequences of images.
34. The system of claim 19 , wherein the plurality of sequences of images comprise images of biological cells.
35. A system comprising:
at least one non-transitory medium encoded with instructions;
at least one processor in communication with the non-transitory medium configured to execute the instructions, wherein when executed, the instructions cause the processor to:
while a value of a loss function stored in a memory is above a minimum value;
write to the memory an initial value of a weight of a feature matrix of the machine learning model:
provide a first plurality of images and a second plurality of images of a plurality of image sequences as a first plurality of inputs to the machine learning model
provide the first plurality of images and a third plurality of images of the plurality of image sequences as a second plurality of inputs to the machine learning model, wherein the first, second, and third plurality of images include noise;
receive a first plurality of outputs responsive to the first plurality of inputs and a second plurality of outputs responsive to the second plurality of inputs
calculate the value of the loss function based, at least in part, on the first plurality of outputs and the second plurality of outputs; and
write an adjusted value of the weight to the memory based, at least in part, on the value of the loss function, wherein the loss function comprises a noise-to-noise term comprising:
L
n
2
n
=
1
N
×
M
∑
k
=
1
N
×
M
{
1
2
out
k
,
i
-
1
-
x
k
,
i
+
1
2
2
+
1
2
out
k
,
i
+
1
-
x
k
,
i
-
1
2
2
-
1
4
out
k
,
i
-
1
-
out
k
,
i
+
1
2
2
}
.
36. The system of claim 35 , wherein N is a number of samples and M is a number of images acquired from each sample.