Systems, methods, and apparatuses for learning semantics-enriched representations via self-discovery, self-classification, and self-restoration in the context of medical imaging
Described herein are means for learning semantics-enriched representations via self-discovery, self-classification, and self-restoration in the context of medical imaging. Embodiments include the training of deep models to learn semantically enriched visual representation by self-discovery, self-classification, and self-restoration of the anatomy underneath medical images, resulting in a collection of semantics-enriched pre-trained models, called Semantic Genesis. Other related embodiments are disclosed.
1. A method performed by a system having at least a processor and a memory therein to execute instructions, wherein the method comprises:
performing a self-discovery operation of anatomical patterns by (i) extracting deep features from each of a plurality of patient scans within a dataset, (ii) training an auto-encoder network utilizing training data, and (iii) selecting a random patient scan from the dataset as a reference image and identifying a set of semantically similar patients to the reference image within the dataset;
performing a self-classification operation of anatomical patterns by encoding input anatomical patterns into a latent space, followed by a sequence of fully-connected (fc) layers to predict a pseudo label associated with each pattern, wherein the anatomical patterns are classified via a categorical cross-entropy loss function; and
performing a self-restoration operation of anatomical patterns by (i) transforming the input anatomical patterns forming transformed anatomical patterns, (ii) encoding the transformed anatomical patterns into a latent space, and (iii) decoding the transformed anatomical patterns to recover the original input anatomical patterns from the transformed anatomical patterns forming reconstructed anatomical patterns.
2. The method of claim 1 , wherein simultaneous processing executes both the self-classification operation and the self-restoration operation in parallel.
3. The method of claim 2 , further comprising:
training a Semantic Genesis model by having the Semantic Genesis model simultaneously classify and restore the transformed anatomical patterns.
4. The method of claim 1 , wherein performing the method works with both 2D and 3D input images.
5. The method of claim 1 , further comprising:
computing an L2 distance between the original input anatomical patterns and the reconstructed anatomical patterns as a loss function.
6. The method of claim 1 , wherein transforming the input anatomical patterns forming transformed anatomical patterns comprises applying one or more transformation functions selected from the group comprising:
(i) a non-linear transformation function,
(ii) a local-shuffling transformation function,
(iii) an out-painting transformation function, and
(iv) an in-painting transformation function.
7. The method of claim 1 , wherein the patient scans within the dataset are not manually annotated.
8. Non-transitory computer-readable storage media having instructions stored thereupon that, when executed by a system having at least a processor and a memory therein, the instructions cause the system to perform operations including:
performing a self-discovery operation of anatomical patterns by (i) extracting deep features from each of a plurality of patient scans within a dataset, (ii) training an auto-encoder network utilizing training data, and (iii) selecting a random patient scan from the dataset as a reference image and identifying a set of semantically similar patients to the reference image within the dataset;
performing a self-classification operation of anatomical patterns by encoding input anatomical patterns into a latent space, followed by a sequence of fully-connected (fc) layers to predict a pseudo label associated with each pattern, wherein the anatomical patterns are classified via a categorical cross-entropy loss function; and
performing a self-restoration operation of anatomical patterns by (i) transforming the input anatomical patterns forming transformed anatomical patterns, (ii) encoding the transformed anatomical patterns into a latent space, and (iii) decoding the transformed anatomical patterns to recover the original input anatomical patterns from the transformed anatomical patterns forming reconstructed anatomical patterns.
9. The non-transitory computer readable storage media of claim 8 , wherein simultaneous processing executes both the self-classification operation and the self-restoration operation in parallel.
10. The non-transitory computer readable storage media of claim 9 , wherein the instructions, when executed by the processor of the system, the instructions cause the system to perform further operations comprising:
training a Semantic Genesis model by having the Semantic Genesis model simultaneously classify and restore the transformed anatomical patterns.
11. The non-transitory computer readable storage media of claim 8 , wherein performing the method works with both 2D and 3D input images.
12. The non-transitory computer readable storage media of claim 8 , wherein the instructions, when executed by the processor of the system, the instructions cause the system to perform further operations comprising:
computing an L2 distance between the original input anatomical patterns and the reconstructed anatomical patterns as a loss function.
13. The non-transitory computer readable storage media of claim 8 , wherein transforming the input anatomical patterns forming transformed anatomical patterns comprises applying one or more transformation functions selected from the group comprising:
(i) a non-linear transformation function,
(ii) a local-shuffling transformation function,
(iii) an out-painting transformation function, and
(iv) an in-painting transformation function.
14. The non-transitory computer readable storage media of claim 8 , wherein the patient scans within the dataset are not manually annotated.
15. A system comprising:
a memory to store instructions;
a processor to execute the instructions stored in the memory;
wherein the system is specially configured to:
execute instructions via the processor for performing a self-discovery operation of anatomical patterns by (i) extracting deep features from each of a plurality of patient scans within a dataset, (ii) training an auto-encoder network utilizing training data, and (iii) selecting a random patient scan from the dataset as a reference image and identifying a set of semantically similar patients to the reference image within the dataset;
execute instructions via the processor for performing a self-classification operation of anatomical patterns by encoding input anatomical patterns into a latent space, followed by a sequence of fully-connected (fc) layers to predict a pseudo label associated with each pattern, wherein the anatomical patterns are classified via a categorical cross-entropy loss function; and
execute instructions via the processor for performing a self-restoration operation of anatomical patterns by (i) transforming the input anatomical patterns forming transformed anatomical patterns, (ii) encoding the transformed anatomical patterns into a latent space, and (iii) decoding the transformed anatomical patterns to recover the original input anatomical patterns from the transformed anatomical patterns forming reconstructed anatomical patterns.
16. The system of claim 15 , wherein simultaneous processing executes both the self-classification operation and the self-restoration operation in parallel; and
wherein the system is further specially configured to execute instructions via the processor for training a Semantic Genesis model by having the Semantic Genesis model simultaneously classify and restore the transformed anatomical patterns.
17. The system of claim 15 , wherein performing the method works with both 2D and 3D input images.
18. The system of claim 15 , wherein the system is further specially configured to:
execute instructions via the processor for computing an L2 distance between the original input anatomical patterns and the reconstructed anatomical patterns as a loss function.
19. The system of claim 15 , wherein transforming the input anatomical patterns forming transformed anatomical patterns comprises applying one or more transformation functions selected from the group comprising:
(i) a non-linear transformation function,
(ii) a local-shuffling transformation function,
(iii) an out-painting transformation function, and
(iv) an in-painting transformation function.
20. The system of claim 15 , wherein the patient scans within the dataset are not manually annotated.