Systems and methods for image segmentation
Described herein are neural network-based systems, methods and instrumentalities associated with image segmentation that may be implementing using an encoder neural network and a decoder neural network. The encoder network may be configured to receive a medical image comprising a visual representation of an anatomical structure and generate a latent representation of the medical image indicating a plurality of features of the medical image. The latent representation may be used by the decoder network to generate a mask for segmenting the anatomical structure from the medical image. The decoder network may be pre-trained to learn a shape prior associated with the anatomical structure and once trained, the decoder network may be used to constrain an output of the encoder network during training of the encoder network.
1. A neural network system implemented using one or more processors, the neural network system comprising:
a first encoder neural network configured to:
receive a medical image comprising a visual representation of an anatomical structure; and
generate a representation of the medical image, the representation indicating a plurality of features extracted from the medical image; and
a decoder neural network configured to:
receive the representation of the medical image from the first encoder neural network; and
generate a mask for segmenting the anatomical structure from the medical image based on the representation of the medical image;
wherein the decoder neural network is trained before the first encoder neural network to learn a shape prior associated with the anatomical structure using a training dataset that represents a shape distribution of the anatomical structure, and wherein, during training of the first encoder neural network, parameters of the decoder neural network are fixed and the decoder neural network is used to constrain an output of the first encoder neural network.
2. The neural network system of claim 1 , wherein the first encoder neural network is trained iteratively and, during a present training iteration of the first encoder neural network:
the decoder neural network is used to generate a segmentation mask associated with a training image based on a representation of the training image generated by the first encoder neural network in a preceding training iteration of the first encoder neural network; and
the first encoder neural network predicts an adjustment to the representation of the training image based on the segmentation mask generated by the decoder neural network and a ground truth associated with the segmentation mask.
3. The neural network system of claim 2 , wherein the representation of the training image comprises a latent space representation of the training image.
4. The neural network system of claim 2 , wherein the training of the first encoder neural network comprises deriving an initial representation of the training image and providing the initial representation of the training image to the decoder neural network so as to obtain an initial prediction of the segmentation mask.
5. The neural network system of claim 2 , wherein multiple instances of the first encoder neural network and the decoder neural network are arranged in a cascading manner in the neural network system.
6. The neural network system of claim 2 , wherein the first encoder neural network comprises a recurrent neural network (RNN).
7. The neural network system of claim 1 , wherein the decoder neural network is co-trained with a second encoder neural network and the co-training comprises:
the second encoder neural network receiving a training image and generating an output to indicate a plurality of features of the training image;
the decoder neural network predicting a segmentation mask associated with the training image based on the output of the second encoder neural network; and
the second encoder neural network and the decoder neural network adjusting their respective parameters based on a loss function associated with the segmentation mask.
8. The neural network system of claim 1 , wherein the decoder neural network is trained without a corresponding encoder neural network and the training of the decoder neural network comprises:
the decoder neural network receiving a latent representation of a training image as an input;
the decoder neural network predicting a segmentation mask associated with the training image based on the latent representation; and
the decoder neural network adjusting its parameters to minimize a loss between the predicted segmentation mask and a ground truth associated with the segmentation mask.
9. The neural network system of claim 1 , wherein the representation of the medical image generated by the first encoder neural network comprises a latent variable.
10. The neural network system of claim 1 , wherein, during the training of the first encoder neural network, the decoder neural network is used to constrain the output of the first encoder neural network based on the shape prior of the anatomical structure learned by the decoder neural network.
11. A method for segmenting a medical image comprising a visual representation of an anatomical structure, the method comprising:
receiving, via a first encoder neural network, the medical image;
generating, via the first encoder neural network, a representation of the medical image, the representation indicating a plurality of features extracted from the medical image;
receiving, at a decoder neural network, the representation of the medical image; and
generating, via the decoder neural network, a mask for segmenting the anatomical structure from the medical image based on the representation of the medical image;
wherein the decoder neural network is trained before the first encoder neural network to learn a shape prior associated with the anatomical structure using training data representing a shape distribution of the anatomical structure, and wherein, during training of the first encoder neural network, parameters of the decoder neural network are fixed and the decoder neural network is used to constrain an output of the first encoder neural network.
12. The method of claim 11 , wherein the first encoder neural network is trained iteratively and, during a present training iteration of the first encoder neural network, the decoder neural network generates a segmentation mask associated with a training image based on a representation of the training image generated by the first encoder neural network in a preceding training iteration of the first encoder neural network and the first encoder neural network predicts an adjustment to the representation of the training image based on the segmentation mask generated by the decoder neural network and a ground truth associated with the segmentation mask.
13. The method of claim 12 , wherein the representation of the training image comprises a latent space representation of the training image.
14. The method of claim 12 , wherein the training of the first encoder neural network comprises deriving an initial representation of the training image and providing the initial representation of the training image to the decoder neural network so as to obtain an initial prediction of the segmentation mask.
15. The method of claim 12 , wherein the mask for segmenting the anatomical structure is generated using multiple instances of the first encoder neural network and the decoder neural network arranged in a cascading manner.
16. The method of claim 12 , wherein the first encoder neural network comprises a recurrent neural network (RNN).
17. The method of claim 11 , wherein the decoder neural network is co-trained with a second encoder neural network and the co-training comprises:
the second encoder neural network receiving a training image and generating an output to indicate a plurality of features of the training image;
the decoder neural network predicting a segmentation mask associated with the training image based on the output of the second encoder neural network; and
the second encoder neural network and the decoder neural network adjusting their respective parameters based on a loss function associated with the segmentation mask.
18. The method of claim 11 , wherein the decoder neural network is trained without a corresponding encoder neural network and the training of the decoder neural network comprises:
the decoder neural network receiving a latent representation of a training image as an input;
the decoder neural network predicting a segmentation mask associated with the training image based on the latent representation; and
the decoder neural network adjusting its parameters to minimize a loss between the predicted segmentation mask and a ground truth associated with the segmentation mask.
19. The method of claim 11 , wherein the representation of the medical image generated by the first encoder neural network comprises a latent variable.
20. A method of training a neural network system for image segmentation, the neural network system comprising a decoder neural network and an encoder neural network, the method comprising:
training the decoder neural network before the encoder neural network to learn a shape prior associated with an anatomical structure, wherein the training of the decoder neural network is performed using a training dataset that represents a shape distribution of the anatomical structure, and wherein the training of the decoder neural network comprises:
providing the decoder neural network with a representation of a medical image comprising a visual representation of the anatomical structure, the representation indicating a plurality of features of the medical image;
causing the decoder neural network to predict a first segmentation mask based on the provided representation; and
causing the decoder neural network to adjust its parameters to minimize a first loss between the first segmentation mask and a ground truth associated with the first segmentation mask; and
training the encoder neural network utilizing the decoder neural network, wherein the encoder neural network is trained iteratively, wherein the parameters of the decoder neural network are fixed during the training of the encoder neural network, and wherein a present training iteration of the encoder neural network comprises:
causing the decoder neural network to predict a second segmentation mask associated with a training image based on a representation of the training image obtained during a preceding training iteration of the encoder neural network;
causing the encoder neural network to update its parameters and predict an adjustment to the representation of the training image based on a difference between the second segmentation mask predicted by the decoder neural network and a ground truth associated with the second segmentation mask; and
obtaining an adjusted representation of the training image based on the adjustment predicted by the encoder neural network and the representation of the training image obtained during the preceding training iteration of the encoder neural network.