Systems, methods, and apparatuses for generating pre-trained models for nnU-net through the use of improved transfer learning techniques
View Patent ↗Described herein are means for generating pre-trained models for nnU-Net through the use of improved transfer learning techniques, in which the pre-trained models are then utilized for the processing of medical imaging. According to a particular embodiment, there is a system specially configured for segmenting medical images, in which such a system includes: a memory to store instructions; a processor to execute the instructions stored in the memory; wherein the system is specially configured to: execute instructions via the processor for executing a pre-trained model from Models Genesis within a nnU-Net framework; execute instructions via the processor for learning generic anatomical patterns within the executing Models Genesis through self-supervised learning; execute instructions via the processor for transforming an original image using distortion and cutout-based methods; execute instructions via the processor for learning the reconstruction of the original image from the transformed image using an encoder-decoder architecture of the nnU-Net framework to identify the generic anatomical representation from the transformed image by recovering the original image; and wherein architecture determined by the nnU-Net framework is utilized with Models Genesis and is trained to minimize the L2 distance between the prediction and ground truth. Other related embodiments are disclosed.
1. A system comprising:
a memory to store instructions;
a set of one or more processors;
a non-transitory machine-readable storage medium that provides instructions that, when executed by the set of one or more processors, the instructions stored in the memory are configurable to cause the system to perform operations comprising:
receiving training data having a plurality of unlabeled images therein;
deriving a pre-trained model from generic autodidactic models which provide a starting point for parameter initialization in a target task within the nnU-Net framework via a self-supervised learning framework which trains 3D models using the unlabeled images by learning appearance, texture and context from the unlabeled images;
executing the pre-trained model within a nnU-Net framework;
learning generic anatomical patterns within the executing generic autodidactic models through self-supervised learning;
transforming an original image into a transformed image using distortion and cutout transformations;
generating a pre-trained model by training the 3D model to learn a reconstruction of the original image from the transformed image using an encoder-decoder architecture of the nnU-Net framework to identify generic anatomical representations from the transformed image by recovering the original image;
executing the trained 3D model to output a prediction of anatomical patterns from a medical image which forms no part of the training data; and
wherein the encoder-decoder architecture determined by the nnU-Net framework is utilized with the generic autodidactic models and is trained to minimize the L2 distance between the prediction and ground truth.
2. The system of claim 1 :
wherein the nnU-Net framework comprises a supervised adaptive framework which applies random parameter initialization within a generic U-Net's encoder-decoder architecture to perform image segmentation for an input image; and
wherein deriving the pre-trained model from generic autodidactic models comprises deriving the pre-trained model from Models Genesis to provide the starting point for parameter initialization in the target task within the nnU-Net framework.
3. The system of claim 1 , wherein the self-supervised learning comprises Non-Linear Transformation in which a Bezier curve is applied as transformation function on an input patch of the original image to enable the model to learn organ appearances and intensity mapping.
4. The system of claim 1 , wherein the self-supervised learning comprises applying local pixel shuffling to sample a random window from an input patch of the original image which are captured as sampled images and then further wherein the local pixel shuffling shuffles the pixels contained in the captured sample images to enable the model to learn textures and edges of organs.
5. The system of claim 1 , wherein the self-supervised learning comprises applying cutout methods including implementing an outer-cutout method and an inner-cutout method to enable the model to learn the context present in an input patch of the original image.
6. The system of claim 5 :
wherein the outer-cutout method includes random windows of different sizes being superimposed together followed by randomly assigning the pixel values outside the window to enable the outer-cutout method to learn the global geometry and spatial layout of the organs in each input patch; and
wherein the inner-cutout method includes assigning pixel values inside the window with a constant value to enable the inner-cutout method to learn the local context of organs in the input patch.
7. The system of claim 1 , wherein learning the reconstruction of the original image comprises training the pre-trained 3D model to learn the reconstruction by performing an image segmentation task to identify human organs or to identify tumors present on the human organs within a new medical image not included within any training data or training images upon which the pre-trained 3D model or the self-supervised learning framework was trained.
8. A computer-implemented method executed by a system having at least a processor and a memory therein, wherein the method comprises:
receiving training data having a plurality of unlabeled images therein;
deriving a pre-trained model from generic autodidactic models which provide a starting point for parameter initialization in a target task within the nnU-Net framework via a self-supervised learning framework which trains 3D models using the unlabeled images by learning appearance, texture and context from the unlabeled images;
executing the pre-trained model within a nnU-Net framework;
learning generic anatomical patterns within the executing generic autodidactic models through self-supervised learning;
transforming an original image into a transformed image using distortion and cutout transformations;
generating a pre-trained model by training the 3D model to learn a reconstruction of the original image from the transformed image using an encoder-decoder architecture of the nnU-Net framework to identify generic anatomical representations from the transformed image by recovering the original image;
executing the trained 3D model to output a prediction of anatomical patterns from a medical image which forms no part of the training data; and
wherein the encoder-decoder architecture determined by the nnU-Net framework is utilized with the generic autodidactic models and is trained to minimize the L2 distance between the prediction and ground truth.
9. The method of claim 8 :
wherein the nnU-Net framework comprises a supervised adaptive framework which applies random parameter initialization within a generic U-Net's encoder-decoder architecture to perform image segmentation for an input image; and
wherein deriving the pre-trained model from generic autodidactic models comprises deriving the pre-trained model from Models Genesis to provide the starting point for parameter initialization in the target task within the nnU-Net framework.
10. The method of claim 8 , wherein the self-supervised learning comprises Non-Linear Transformation in which a Bezier curve is applied as transformation function on an input patch of the original image to enable the model to learn organ appearances and intensity mapping.
11. The method of claim 8 , wherein the self-supervised learning comprises applying local pixel shuffling to sample a random window from an input patch of the original image which are captured as sampled images and then further wherein the local pixel shuffling shuffles the pixels contained in the captured sample images to enable the model to learn textures and edges of organs.
12. The method of claim 8 , wherein the self-supervised learning comprises applying cutout methods including implementing an outer-cutout method and an inner-cutout method to enable the model to learn the context present in an input patch of the original image.
13. The method of claim 12 :
wherein the outer-cutout method includes random windows of different sizes being superimposed together followed by randomly assigning the pixel values outside the window to enable the outer-cutout method to learn the global geometry and spatial layout of the organs in each input patch; and
wherein the inner-cutout method includes assigning pixel values inside the window with a constant value to enable the inner-cutout method to learn the local context of organs in the input patch.
14. The method of claim 8 , wherein learning the reconstruction of the original image comprises training the pre-trained 3D model to learn the reconstruction by performing an image segmentation task to identify human organs or to identify tumors present on the human organs within a new medical image not included within any training data or training images upon which the pre-trained 3D model or the self-supervised learning framework was trained.
15. Non-transitory computer readable storage media having instructions stored thereupon that, when executed by a system having at least a processor and a memory therein, the instructions cause the processor to perform operations including:
receiving training data having a plurality of unlabeled images therein;
deriving a pre-trained model from generic autodidactic models which provide a starting point for parameter initialization in a target task within the nnU-Net framework via a self-supervised learning framework which trains 3D models using the unlabeled images by learning appearance, texture and context from the unlabeled images;
executing the pre-trained model within a nnU-Net framework;
learning generic anatomical patterns within the executing generic autodidactic models through self-supervised learning;
transforming an original image into a transformed image using distortion and cutout transformations;
generating a pre-trained model by training the 3D model to learn a reconstruction of the original image from the transformed image using an encoder-decoder architecture of the nnU-Net framework to identify generic anatomical representations from the transformed image by recovering the original image;
executing the trained 3D model to output a prediction of anatomical patterns from a medical image which forms no part of the training data; and
wherein the encoder-decoder architecture determined by the nnU-Net framework is utilized with the generic autodidactic models and is trained to minimize the L2 distance between the prediction and ground truth.
16. The non-transitory computer readable storage media of claim 15 , wherein the self-supervised learning comprises Non-Linear Transformation in which a Bezier curve is applied as transformation function on an input patch of the original image to enable the model to learn organ appearances and intensity mapping.
17. The non-transitory computer readable storage media of claim 15 , wherein the self-supervised learning comprises applying local pixel shuffling to sample a random window from an input patch of the original image which are captured as sampled images and then further wherein the local pixel shuffling shuffles the pixels contained in the captured sample images to enable the model to learn textures and edges of organs.
18. The non-transitory computer readable storage media of claim 15 , wherein the self-supervised learning comprises applying cutout methods including implementing an outer-cutout method and an inner-cutout method to enable the model to learn the context present in an input patch of the original image.
19. The non-transitory computer readable storage media of claim 17 :
wherein the outer-cutout method includes random windows of different sizes being superimposed together followed by randomly assigning the pixel values outside the window to enable the outer-cutout method to learn the global geometry and spatial layout of the organs in each input patch; and
wherein the inner-cutout method includes assigning pixel values inside the window with a constant value to enable the inner-cutout method to learn the local context of organs in the input patch.
20. The non-transitory computer readable storage media of claim 15 , wherein learning the reconstruction of the original image comprises training the pre-trained 3D model to learn the reconstruction by performing an image segmentation task to identify human organs or to identify tumors present on the human organs within a new medical image not included within any training data or training images upon which the pre-trained 3D model or the self-supervised learning framework was trained.