Methods and systems for automated image segmentation of anatomical structure
This disclosure relates generally to methods and systems for automated image segmentation of an anatomical structure such as heart. Most of the techniques in literature are using 2-D or slice by-slice data due to lightweight and need of less data for training. These networks lack 3-D contextual information. Further, the conventional techniques are inaccurate and inefficient in the 3-D image segmentation till the last slice of the image. The present disclosure solves automated 3-D image segmentation of the anatomical structure such as heart, by proposing a new Generative Adversarial Network (GAN) based architecture for the 3-D segmentation, with a patch-based extraction technique and a class-weighted generalized dice loss. The proposed 3-D GAN based architecture is capable of storing the 3-D contextual information for the image segmentation of the anatomical structure, with high accuracy.
1 . A processor-implemented method for automated image segmentation of an anatomical structure, comprising the steps of:
receiving, via one or more hardware processors, a plurality of 3-dimensional (3-D) training images corresponding to the anatomical structure and a ground-truth 3-D image associated with each of the plurality of 3-D training images, wherein the plurality of 3-D training images is associated with a plurality of classes of the anatomical structure;
pre-processing, via the one or more hardware processors, the plurality of 3-D training images, to obtain a plurality of pre-processed training images;
forming, via the one or more hardware processors, one or more mini-batches from the plurality of pre-processed training images, based on a predefined mini-batch size, wherein each mini-batch comprises one or more pre-processed training images; and
training, via the one or more hardware processors, a segmentation network model, with the one or more pre-processed training images present in each mini-batch at a time, until the one or more mini-batches are completed for a predefined training epochs, to obtain a trained segmentation network model, wherein the segmentation network model comprises a generator and a patch-based discriminator, and training the segmentation network model with the one or more pre-processed training images present in each mini-batch comprises:
passing each pre-processed training image present in the mini-batch to an encoder network of the generator, to obtain a set of patched feature maps and a set of encoded feature maps, corresponding to the pre-processed training image;
channel-wise concatenating the set of patched feature maps and the set of encoded feature maps, through a bottleneck network of the generator, to obtain a concatenated feature map corresponding to each pre-processed training image;
passing the concatenated feature map to a decoder network of the generator, to predict a segmented image corresponding to each pre-processed training image;
predicting a probability value corresponding to each pre-processed training image, by using (i) the predicted segmented image corresponding to the pre-processed training image and (ii) the ground-truth 3-D image of the corresponding pre-processed training image, through the patch-based discriminator;
calculating a value of a loss function of the segmentation network model, for the one or more pre-processed training images present in the mini-batch, using the predicted probability value corresponding to each pre-processed training image; and
backpropagating weights of the segmentation network model, based on the calculated value of the loss function of the segmentation network model.
2 . The processor-implemented method of claim 1 , further comprising:
receiving, via the one or more hardware processors, an input 3-D training image corresponding to the anatomical structure for the segmentation; and
passing, via the one or more hardware processors, the input 3-D training image, to the trained segmentation model, to predict the segmented image corresponding to the input 3-D training image of the anatomical structure.
3 . The processor-implemented method of claim 2 , further comprising:
obtaining, via the one or more hardware processors, one or more domain features of the predicted segmented image corresponding to the input 3-dimensional training image, using a domain feature extraction technique; and
passing, via the one or more hardware processors, the one or more domain features of the predicted segmented image, to a classification network model, to predict a classification having a plurality of anomaly classes.
4 . The processor-implemented method of claim 1 , wherein pre-processing each 3-dimensional training image, to obtain a corresponding pre-processed training image, comprising: sequentially performing at least one of: (i) an image orientation normalization, (ii) a region of interest (ROI) extraction, (iii) a size normalization, (iv) a pixel value normalization, and (v) an image data augmentation, on each 3-D training image.
5 . The processor-implemented method of claim 1 , wherein the segmentation network model is a generative adversarial network (GAN) and comprising the generator and the patch-based discriminator, and wherein the generator comprises the encoder network, the bottleneck network, the decoder network, and a set of skip connections between the encoder network and the decoder network.
6 . The processor-implemented method of claim 1 , wherein the loss function of the segmentation network model for each pre-processed training image, comprises a generator loss and a discriminator loss, wherein the generator loss comprises a class-weighted generalized dice loss and an adversarial loss, and the discriminator loss comprises a real loss and a fake loss, and wherein:
the class-weighted generalized dice loss is calculated between the ground-truth 3-D image of the corresponding pre-processed training image and the predicted segmented image corresponding to the pre-processed training image, wherein the class-weighted generalized dice loss is calculated using pixel-based distribution technique;
the adversarial loss is calculated between the ground-truth 3-D image of the corresponding pre-processed training image and the predicted segmented image corresponding to the pre-processed training image;
the real loss is calculated between the corresponding pre-processed training image and the ground-truth 3-D image of the corresponding pre-processed training image; and
the fake loss is calculated between the corresponding pre-processed training image and predicted segmented image corresponding to the pre-processed training image.
7 . The processor-implemented method of claim 6 , wherein the class-weighted generalized dice loss is defined with one or more class weights that are associated with plurality of classes of the anatomical structure.
8 . The processor-implemented method of claim 1 , wherein a learning rate and a dropout of the segmentation network model are dynamically adjusted between the predefined training epochs during the training, based on the value of loss function at each predefined training epoch.
9 . A system for automated image segmentation of an anatomical structure, comprising:
a memory storing instructions;
one or more input/output (I/O) interfaces; and
one or more hardware processors coupled to the memory via the one or more I/O interfaces, wherein the one or more hardware processors are configured by the instructions to:
receive a plurality of 3-dimensional (3-D) training images corresponding to the anatomical structure and a ground-truth 3-D image associated with each of the plurality of 3-D training images, wherein the plurality of 3-D training images is associated with a plurality of classes of the anatomical structure;
pre-process the plurality of 3-D training images, to obtain a plurality of pre-processed training images;
form one or more mini-batches from the plurality of pre-processed training images, based on a predefined mini-batch size, wherein each mini-batch comprises one or more pre-processed training images; and
train a segmentation network model, with the one or more pre-processed training images present in each mini-batch at a time, until the one or more mini-batches are completed for a predefined training epochs, to obtain a trained segmentation network model, wherein the segmentation network model comprises a generator and a patch-based discriminator, and training the segmentation network model) with the one or more pre-processed training images present in each mini-batch comprises:
passing each pre-processed training image present in the mini-batch to an encoder network of the generator, to obtain a set of patched feature maps and a set of encoded feature maps, corresponding to the pre-processed training image;
channel-wise concatenating the set of patched feature maps and the set of encoded feature maps, through a bottleneck network of the generator, to obtain a concatenated feature map corresponding to each pre-processed training image;
passing the concatenated feature map to a decoder network of the generator, to predict a segmented image corresponding to each pre-processed training image;
predicting a probability value corresponding to each pre-processed training image, by using (i) the predicted segmented image corresponding to the pre-processed training image and (ii) the ground-truth 3-D image of the corresponding pre-processed training image, through the patch-based discriminator;
calculating a value of a loss function of the segmentation network model, for the one or more pre-processed training images present in the mini-batch, using the predicted probability value corresponding to each pre-processed training image; and
backpropagating weights of the segmentation network model, based on the calculated value of the loss function of the segmentation network model.
10 . The system of claim 9 , wherein the one or more hardware processors are further configured by the instructions to:
receive an input 3-D training image corresponding to the anatomical structure for the segmentation; and
pass the input 3-D training image, to the trained segmentation model, to predict the segmented image corresponding to the input 3-D training image of the anatomical structure.
11 . The system of claim 10 , wherein the one or more hardware processors are further configured by the instructions to:
obtain one or more domain features of the predicted segmented image corresponding to the input 3-dimensional training image, using a domain feature extraction technique; and
pass the one or more domain features of the predicted segmented image, to a classification network model, to predict a classification having a plurality of anomaly classes.
12 . The system of claim 9 , wherein the one or more hardware processors are configured to pre-process each 3-dimensional training image, to obtain a corresponding pre-processed training image, by: sequentially performing at least one of: (i) an image orientation normalization, (ii) a region of interest (ROI) extraction, (iii) a size normalization, (iv) a pixel value normalization, and (v) an image data augmentation, on each 3-D training image.
13 . The system of claim 9 , wherein the segmentation network model is a generative adversarial network (GAN) and comprising the generator and the patch-based discriminator, and wherein the generator comprises the encoder network, the bottleneck network, the decoder network, and a set of skip connections between the encoder network and the decoder network.
14 . The system of claim 9 , wherein the loss function of the segmentation network model for each pre-processed training image, comprises a generator loss and a discriminator loss, wherein the generator loss comprises a class-weighted generalized dice loss and an adversarial loss, and the discriminator loss comprises a real loss and a fake loss, and wherein:
the class-weighted generalized dice loss is calculated between the ground-truth 3-D image of the corresponding pre-processed training image and the predicted segmented image corresponding to the pre-processed training image, wherein the class-weighted generalized dice loss is calculated using pixel-based distribution technique;
the adversarial loss is calculated between the ground-truth 3-D image of the corresponding pre-processed training image and the predicted segmented image corresponding to the pre-processed training image;
the real loss is calculated between the corresponding pre-processed training image and the ground-truth 3-D image of the corresponding pre-processed training image; and
the fake loss is calculated between the corresponding pre-processed training image and predicted segmented image corresponding to the pre-processed training image.
15 . The system of claim 14 , wherein the class-weighted generalized dice loss is defined with one or more class weights that are associated with plurality of classes of the anatomical structure.
16 . The system of claim 9 , wherein the one or more hardware processors are configured to dynamically adjust a learning rate and a dropout of the segmentation network model between the predefined training epochs during the training, based on the value of loss function at each predefined training epoch.
17 . One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:
receiving a plurality of 3-dimensional (3-D) training images corresponding to the anatomical structure and a ground-truth 3-D image associated with each of the plurality of 3-D training images, wherein the plurality of 3-D training images is associated with a plurality of classes of the anatomical structure;
pre-processing the plurality of 3-D training images, to obtain a plurality of pre-processed training images;
forming one or more mini-batches from the plurality of pre-processed training images, based on a predefined mini-batch size, wherein each mini-batch comprises one or more pre-processed training images; and
training a segmentation network model, with the one or more pre-processed training images present in each mini-batch at a time, until the one or more mini-batches are completed for a predefined training epochs, to obtain a trained segmentation network model, wherein the segmentation network model comprises a generator and a patch-based discriminator, and training the segmentation network model) with the one or more pre-processed training images present in each mini-batch comprises:
passing each pre-processed training image present in the mini-batch to an encoder network of the generator, to obtain a set of patched feature maps and a set of encoded feature maps, corresponding to the pre-processed training image;
channel-wise concatenating the set of patched feature maps and the set of encoded feature maps, through a bottleneck network of the generator, to obtain a concatenated feature map corresponding to each pre-processed training image;
passing the concatenated feature map to a decoder network of the generator, to predict a segmented image corresponding to each pre-processed training image;
predicting a probability value corresponding to each pre-processed training image, by using (i) the predicted segmented image corresponding to the pre-processed training image and (ii) the ground-truth 3-D image of the corresponding pre-processed training image, through the patch-based discriminator;
calculating a value of a loss function of the segmentation network model, for the one or more pre-processed training images present in the mini-batch, using the predicted probability value corresponding to each pre-processed training image; and
backpropagating weights of the segmentation network model, based on the calculated value of the loss function of the segmentation network model;
receiving an input 3-D training image corresponding to the anatomical structure for the segmentation;
passing the input 3-D training image, to the trained segmentation model, to predict the segmented image corresponding to the input 3-D training image of the anatomical structure;
obtaining one or more domain features of the predicted segmented image corresponding to the input 3-dimensional training image, using a domain feature extraction technique; and
passing the one or more domain features of the predicted segmented image, to a classification network model, to predict a classification having a plurality of anomaly classes.