Systems and methods for reconstructing images using uncertainty loss
Model-based image reconstruction (MBIR) methods using convolutional neural networks (CNNs) as priors have demonstrated superior image quality and robustness compared to conventional methods. Studies have explored MBIR combined with supervised and unsupervised denoising techniques for image reconstruction in magnetic resonance imaging (MRI) and positron emission tomography (PET). Unsupervised methods like the deep image prior (DIP) have shown promising results and are less prone to hallucinations. However, since the noisy image is used as a reference, strategies to prevent overfitting are unclear. Recently, Bayesian DIP (BDIP) networks that model uncertainty tend to prevent overfitting without requiring early stopping. However, BDIP has not been studied with data-fidelity term for image reconstruction. Present disclosure provides systems and method that implement a MBIR framework with a modified BDIP. Specifically, an uncertainty-based penalty is included to the BDIP to improve reconstruction across iterations.
1 . A processor implemented method, comprising:
receiving, via one or more hardware processors, one or more measurements obtained from a pre-defined forward model associated with a physical process; and
constructing, via the one or more hardware processors, a first image based on the one or more measurements;
training, a neural network via the one or more hardware processors, using the first image and a second image to obtain a third image and a fourth image, wherein the first image serves as an input to a loss function for training the neural network, wherein the second image serves as an input to the neural network, wherein the third image is predicted image or reconstructed image and the fourth image is variance image;
iteratively performing:
performing a comparison of the first image with the third image to obtain a training loss comprising a reconstruction loss, wherein the training loss is fed back to the neural network for training the neural network, wherein the training loss comprises the reconstruction loss when performing a first iteration and the training loss comprises the reconstruction loss and an uncertainty loss when performing subsequent iteration, wherein the training is pre-configured or dynamically determined, wherein the third image and the fourth image are outputted/predicted at a final layer of the neural network being trained;
estimating the uncertainty loss using the fourth image;
training the neural network via the one or more hardware processors, using the third image and the fourth image of a previous iteration, the training loss comprising the reconstruction loss, and the uncertainty loss to obtain the third image and the fourth image of the current iteration; and
performing a comparison of the fourth image of (i) the current iteration and (ii) the previous iteration,
until (i) a difference in an output of the fourth image of (a) the current iteration
and (b) the previous iteration reaches a threshold, or
(ii) a pre-defined number of iterations.
2 . The processor implemented method of claim 1 , wherein the first image is a positron emission tomography (PET) image, wherein the PET image is a noisy image.
3 . The processor implemented method of claim 1 , wherein the second image is a magnetic resonance imaging (MRI) image.
4 . The processor implemented method of claim 1 , further comprising training one or more weights of the neural network using the training loss.
5 . The processor implemented method of claim 1 , wherein a weighted loss function is computed based on the first image, the third image and the fourth image.
6 . The processor implemented method of claim 1 , wherein the third image and the fourth image are obtained by splitting a final layer of the neural network.
7 . A system, comprising:
a memory storing instructions;
one or more communication interfaces; and
one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
receive one or more measurements obtained from a pre-defined forward model associated with a physical process; and
constructing a first image based on the one or more measurements;
training a neural network using the first image and a second image to obtain a third image and a fourth image, wherein the first image serves as an input to a loss function for training the neural network, wherein the second image serves as an input to the neural network, wherein the third image is predicted image or reconstructed image and the fourth image is variance image;
iteratively perform:
performing a comparison of the first image with the third image to obtain a training loss comprising a reconstruction loss, wherein the training loss is fed back to the neural network for training the neural network, wherein the training loss comprises the reconstruction loss when performing a first iteration and the training loss comprises the reconstruction loss and an uncertainty loss when performing subsequent iteration, wherein the training is pre-configured or dynamically determined, wherein the third image and the fourth image are outputted/predicted at a final layer of the neural network being trained;
estimating the uncertainty loss using the fourth image;
training the neural network via the one or more hardware processors, using the third image and the fourth image of a previous iteration, the training loss comprising the reconstruction loss, and the uncertainty loss to obtain the third image and the fourth image of the current iteration; and
performing a comparison of the fourth image of (i) the current iteration and (ii) the previous iteration,
until (i) a difference in an output of the fourth image of (a) the current iteration and (b) the previous iteration reaches a threshold, or (ii) a pre-defined number of iterations.
8 . The system of claim 7 , wherein the first image is a positron emission tomography (PET) image, wherein the PET image is a noisy image.
9 . The system of claim 7 , wherein the second image is a magnetic resonance imaging (MRI) image.
10 . The system of claim 7 , wherein the one or more hardware processors are further configured by the instructions to train one or more weights of the neural network using the training loss.
11 . The system of claim 7 , wherein a weighted loss function is computed based on the first image, the third image and the fourth image.
12 . The system of claim 7 , wherein the third image and the fourth image are obtained by splitting a final layer of the neural network.
13 . One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:
receiving one or more measurements obtained from a pre-defined forward model associated with a physical process; and
constructing a first image based on the one or more measurements;
training, a neural network, using the first image and a second image to obtain a third image and a fourth image, wherein the first image serves as an input to a loss function for training the neural network, wherein the second image serves as an input to the neural network, wherein the third image is predicted image or reconstructed image and the fourth image is variance image;
iteratively performing:
performing a comparison of the first image with the third image to obtain a training loss comprising a reconstruction loss, wherein the training loss is fed back to the neural network for training the neural network, wherein the training loss comprises the reconstruction loss when performing a first iteration and the training loss comprises the reconstruction loss and an uncertainty loss when performing subsequent iteration, wherein the training is pre-configured or dynamically determined, wherein the third image and the fourth image are outputted/predicted at a final layer of the neural network being trained;
estimating the uncertainty loss using the fourth image;
training the neural network, using the third image and fourth image of a previous iteration, the training loss comprising the reconstruction loss, and the uncertainty loss to obtain the third image and the fourth image of the current iteration; and
performing a comparison of the fourth image of (i) the current iteration and (ii) the previous iteration,
until (i) a difference in an output of the fourth image of (a) the current iteration and (b) the previous iteration reaches a threshold, or (ii) a pre-defined number of iterations.
14 . The one or more non-transitory machine-readable information storage mediums of claim 13 , wherein the first image is a positron emission tomography (PET) image, wherein the PET image is a noisy image.
15 . The one or more non-transitory machine-readable information storage mediums of claim 13 , wherein the second image is a magnetic resonance imaging (MRI) image.
16 . The one or more non-transitory machine-readable information storage mediums of claim 13 , wherein the one or more instructions which when executed by the one or more hardware processors further cause training one or more weights of the neural network using the training loss.
17 . The one or more non-transitory machine-readable information storage mediums of claim 13 , wherein a weighted loss function is computed based on the first image, the third image and the fourth image.
18 . The one or more non-transitory machine-readable information storage mediums of claim 13 , wherein the third image and the fourth image are obtained by splitting a final layer of the neural network.