Systems, methods, and apparatuses for implementing contrastive learning via reconstruction within a self-supervised learning framework
View Patent ↗Described herein are means for implementing contrastive learning via reconstruction within a self-supervised learning framework, in which the trained deep models are then utilized for the processing of medical imaging. For instance, an exemplary system is specially configured for performing a random cropping operation to crop a 3D cube from each of a plurality of medical images received at the system as input; performing a resize operation of the cropped 3D cubes; performing an image reconstruction operation of the resized and cropped 3D cubes to predict the whole image represented by the original medical images received; and generating a reconstructed image which is analyzed for reconstruction loss against the original image representing a known ground truth image to the reconstruction loss function. Other related embodiments are disclosed.
1. A system comprising:
a memory to store instructions;
a processor to execute the instructions stored in the memory;
wherein the system is specially configured to execute the instructions via the processor to:
receive at the system as input a plurality of whole three-dimensional (3D) medical images;
for each of the plurality of whole 3D medical images:
randomly crop each of a plurality of 3D parts from the whole 3D medical image received at the system as input;
resize each of the plurality of randomly cropped 3D parts;
predict, based on each of the randomly cropped and resized plurality of 3D parts, the whole 3D medical image;
reconstruct the whole 3D medical image from the predicted whole 3D medical image;
measure a reconstruction loss, according to a reconstruction loss function, in the reconstructed whole 3D medical image compared to the whole 3D medical image; and
analyze the reconstruction loss against the whole 3D medical image representing a known ground truth image to the reconstruction loss function.
2. The system of claim 1 , wherein to randomly crop each of the plurality of 3D parts from the whole 3D medical image comprises to randomly crop, using random scales and random aspect ratios, each of the plurality of 3D parts from the whole 3D medical image.
3. The system of claim 2 , wherein the random scales and random aspect ratios erase low-level cues across, but maintain informative structures and textures amongst, different ones of the randomly cropped plurality of 3D parts.
4. The system of claim 1 , wherein to resize each of the plurality of randomly cropped 3D parts comprises to resize each of the plurality of randomly cropped 3D parts to produce transformed 3D parts; and
wherein to predict, based on each of the randomly cropped and resized plurality of 3D parts, the whole 3D medical image, comprises to predict, based on each of the transformed 3D parts, the whole 3D medical image.
5. The system of claim 4 , wherein to predict, based on each of the transformed 3D parts, the whole 3D medical image comprises to train an encoder-decoder pair to minimize the reconstruction loss, measured according to the reconstruction loss function, between each of the transformed 3D parts and the whole 3D medical image.
6. The system of claim 1 , further comprising a Convolutional Neural Network (CNN) including:
a decoder to output the reconstructed whole 3D medical images; and
an encoder coupled to the decoder to learn contrastive representations that embed part-whole semantics based on the reconstructed whole 3D medical images.
7. The system of claim 6 , wherein the CNN comprises a U-Net architecture; and
the coupling between the encoder and the decoder excludes a skip connection between the encoder and decoder in the U-Net architecture.
8. The system of claim 7 , wherein the skip connection excluded between the encoder and the decoder in the U-Net architecture comprises the skip connection excluded between the encoder and the decoder in the U-Net architecture during training of the CNN to learn the contrastive representations, to force a plurality of bottleneck representations that encode high-level information in the randomly cropped plurality of 3D parts.
9. The system of claim 1 , wherein to resize each of the plurality of randomly cropped 3D parts comprises to resize each of the plurality of randomly cropped 3D parts according to a configurable part size to avoid training an auto-encoder without learning semantics when the part size is too large and to avoid an unsolvable task when the part size is too small so as to lack sufficient information.
10. A method performed by a system having at least a processor and a memory therein to execute instructions, wherein the method comprises:
receiving at the system as input a plurality of whole three-dimensional (3D) medical images;
for each of the plurality of whole 3D medical images:
randomly cropping, using random scales and random aspect ratios, each of a plurality of 3D parts from the whole 3D medical image received at the system as input;
resizing each of the plurality of randomly cropped 3D parts;
predicting, based on each of the randomly cropped and resized plurality of 3D parts, the whole 3D medical image;
reconstructing the whole 3D medical image from the predicted whole 3D medical image;
measuring a reconstruction loss, according to a reconstruction loss function, in the reconstructed whole 3D medical image compared to the whole 3D medical image; and
analyzing the reconstruction loss against the whole 3D medical image representing a known ground truth image to the reconstruction loss function.
11. The method of claim 10 , wherein randomly cropping each of the plurality of 3D parts from the whole 3D medical image comprises randomly cropping, using random scales and random aspect ratios, each of the plurality of 3D parts from the whole 3D medical image.
12. The method of claim 11 , wherein the random scales and random aspect ratios erase low-level cues across, but maintain informative structures and textures amongst, different ones of the randomly cropped plurality of 3D parts.
13. The method of claim 10 , wherein resizing each of the plurality of randomly cropped 3D parts comprises resizing each of the plurality of randomly cropped 3D parts to produce transformed 3D parts; and
wherein to predict, based on each of the randomly cropped and resized plurality of 3D parts, the whole 3D medical image, comprises to predict, based on each of the transformed 3D parts, the whole 3D medical image.
14. The method of claim 13 , wherein predicting, based on each of the transformed 3D parts, the whole 3D medical image comprises training an encoder-decoder pair to minimize the reconstruction loss, measured according to the reconstruction loss function between each of the transformed 3D parts and the whole 3D medical image.
15. The method of claim 10 , further comprising a decoder of a Convolutional Neural Network (CNN) providing as output the reconstructed whole 3D medical images; and
an encoder coupled to the decoder learning contrastive representations that embed part-whole semantics based on the reconstructed whole 3D medical images.
16. The method of claim 15 , wherein the CNN comprises a U-Net architecture; and
the coupling between the encoder and the decoder excluding a skip connection between the encoder and the decoder in the U-Net architecture.
17. The method of claim 16 , wherein the skip connection excluded between the encoder and the decoder in the U-Net architecture comprises the skip connection excluded between the encoder and the decoder in the U-Net architecture during training of the CNN to learn the contrastive representations, to force a plurality of bottleneck representations encoding high-level information in the randomly cropped plurality of 3D parts.
18. The method of claim 10 , wherein resizing each of the plurality of randomly cropped 3D parts comprises resizing each of the plurality of randomly cropped parts according to a configurable part size to avoid training an auto-encoder without learning semantics when the part size is too large and to avoid an unsolvable task when the part size is too small so as to lack sufficient information.
19. A non-transitory computer-readable storage media having instructions stored thereupon that, when executed by a system having at least a processor and a memory therein, cause the system to:
receive at the system as input a plurality of whole three-dimensional (3D) medical images;
randomly crop, using random scales and random aspect ratios, each of a plurality of 3D parts from the whole 3D medical image received at the system as input;
resize each of the plurality of randomly cropped 3D parts;
predict, based on each of the randomly cropped and resized plurality of 3D parts, the whole 3D medical image;
reconstruct the whole 3D medical image from the predicted whole 3D medical image;
measure a reconstruction loss, according to a reconstruction loss function, in the reconstructed whole 3D medical image compared to the whole 3D medical image; and
analyze the reconstruction loss against the whole 3D medical image representing a known ground truth image to the reconstruction loss function.
20. The non-transitory computer-readable storage media of claim 19 , wherein the random scales and random aspect ratios erase low-level cues across, but maintain informative structures and textures amongst, different ones of the randomly cropped plurality of 3D parts.