Systems, methods, and apparatuses for training a deep model to learn contrastive representations embedded within part-whole semantics via a self-supervised learning framework
Described herein are means for training a deep model to learn contrastive representations embedded within part-whole semantics via a self-supervised learning framework, in which the trained deep models are then utilized for the processing of medical imaging. For instance, an exemplary system is specifically configured for performing a random cropping operation to crop a 3D cube from each of a plurality of medical images received at the system as input; performing a resize operation of the cropped 3D cubes; performing an image reconstruction operation of the resized and cropped 3D cubes to predict the resized whole image represented by the original medical images received; and generating a reconstructed image which is analyzed for reconstruction loss against the original image representing a known ground truth image to the reconstruction loss function. Other related embodiments are disclosed.
1. A system comprising:
a memory to store instructions;
a processor to execute the instructions stored in the memory;
wherein the system is specially configured to learn contrastive representations embedded within part-whole semantics via a self-supervised learning framework, by performing the following operations:
performing a random cropping operation to crop a 3D cube from each of a plurality of medical images received at the system as input;
performing a resize operation of the cropped 3D cubes;
performing an image reconstruction operation of the resized and cropped 3D cubes to predict the resized whole image represented by the original medical images received; and
generating a reconstructed image which is analyzed for reconstruction loss against the original image representing a known ground truth image to the reconstruction loss function.
2. The system of claim 1 , wherein randomly cropping the 3D cube comprises cropping the 3D cube utilizing random scales and random aspect ratios.
3. The system of claim 2 , wherein the random scales and random aspect ratios utilized for the random cropping erase low-level cues across different parts but maintain informative structures and textures amongst the randomly cropped 3D cubes.
4. The system of claim 1 , wherein resizing the cropped 3D cubes comprises resizing the cropped 3D cubes to produce transformed part for later reconstruction.
5. The system of claim 1 , wherein the reconstruction is to predict the resized whole image from a local patch by training an encoder-decoder pair to minimize the loss function between the transformed part produced via the random cropping and resizing and the original whole image.
6. The system of claim 5 , wherein the encoder learns contrastive representations that embed the part-whole semantics.
7. The system of claim 5 , wherein all skip connections connecting the encoder and decoder are removed from a U-Net architecture.
8. The system of claim 7 , wherein the skip connections remain absent during training so as to force a bottleneck representation encoding high-level information.
9. The system of claim 1 , wherein a part size via which to resize the cropped 3D cube is configurable to avoid training an auto encoder without learning semantics when the part size is too large and to avoid an unsolvable task when the part size is too small so as to lack sufficient information.
10. Non-transitory computer-readable storage media having instructions stored thereupon that, when executed by a system having at least a processor and a memory therein, the instructions cause the system to learn contrastive representations embedded within part-whole semantics via a self-supervised learning framework, by performing operations including:
performing a random cropping operation to crop a 3D cube from each of a plurality of medical images received at the system as input;
performing a resize operation of the cropped 3D cubes;
performing an image reconstruction operation of the resized and cropped 3D cubes to predict the resized whole image represented by the original medical images received; and
generating a reconstructed image which is analyzed for reconstruction loss against the original image representing a known ground truth image to the reconstruction loss function.
11. The non-transitory computer readable storage media of claim 10 :
wherein randomly cropping the 3D cube comprises cropping the 3D cube utilizing random scales and random aspect ratios; and
wherein the random scales and random aspect ratios utilized for the random cropping erase low-level cues across different parts but maintain informative structures and textures amongst the randomly cropped 3D cubes.
12. The non-transitory computer readable storage media of claim 10 :
wherein resizing the cropped 3D cubes comprises resizing the cropped 3D cubes to produce transformed part for later reconstruction.
13. The non-transitory computer readable storage media of claim 10 :
wherein the reconstruction is to predict the resized whole image from a local patch by training an encoder-decoder pair to minimize the loss function between the transformed part produced via the random cropping and resizing and the original whole image.
14. The non-transitory computer readable storage media of claim 13 :
wherein the encoder learns contrastive representations that embed the part-whole semantics.
15. The non-transitory computer readable storage media of claim 13 :
wherein all skip connections connecting the encoder and decoder are removed from a U-Net architecture; and wherein the skip connections remain absent during training so as to force a bottleneck representation encoding high-level information.
16. The non-transitory computer readable storage media of claim 10 :
wherein a part size via which to resize the cropped 3D cube is configurable to avoid training an auto encoder without learning semantics when the part size is too large and to avoid an unsolvable task when the part size is too small so as to lack sufficient information.
17. A method performed by a system having at least a processor and a memory therein to execute instructions for learning contrastive representations embedded within part-whole semantics via a self-supervised learning framework, wherein the method comprises:
performing a random cropping operation to crop a 3D cube from each of a plurality of medical images received at the system as input;
performing a resize operation of the cropped 3D cubes;
performing an image reconstruction operation of the resized and cropped 3D cubes to predict the resized whole image represented by the original medical images received; and
generating a reconstructed image which is analyzed for reconstruction loss against the original image representing a known ground truth image to the reconstruction loss function.
18. The method of claim 17 :
wherein randomly cropping the 3D cube comprises cropping the 3D cube utilizing random scales and random aspect ratios; and
wherein the random scales and random aspect ratios utilized for the random cropping erase low-level cues across different parts but maintain informative structures and textures amongst the randomly cropped 3D cubes.
19. The method of claim 17 :
wherein the reconstruction is to predict the resized whole image from a local patch by training an encoder-decoder pair to minimize the loss function between the transformed part produced via the random cropping and resizing and the original whole image.
20. The method of claim 19 : wherein all skip connections connecting the encoder and decoder are removed from a U-Net architecture; and wherein the skip connections remain absent during training so as to force a bottleneck representation encoding high-level information.