Self-supervised learning for modeling a 3D brain anatomical representation
Systems and methods for performing a medical imaging analysis task are provided. A plurality of 3D (three dimensional) patches extracted from a 3D input medical image is received. A set of local features is extracted from each of the plurality of 3D patches using a machine learning based local feature extractor network. Global features representing relationships between the sets of local features are determined. A medical imaging analysis task is performed on the 3D input medical image based on the global features. Results of the medical imaging analysis task are output.
1 . A computer-implemented method comprising:
receiving a plurality of 3D (three dimensional) patches extracted from a 3D input medical image;
extracting a set of local features from each of the plurality of 3D patches using a machine learning based local feature extractor network;
reconstructing, using a machine learning based decoder network, the sets of local features output by a bottleneck layer of the machine learning based feature extractor network;
determining global features representing relationships between the sets of local features;
hierarchically combining the global features with the reconstructed sets of local features at a plurality of scales;
performing a medical imaging analysis task on the 3D input medical image based on the hierarchical combination of the global features and the decoded sets of local features; and
outputting results of the medical imaging analysis task.
2 . The computer-implemented method of claim 1 , wherein the plurality of 3D patches is of a fixed resolution and a fixed field-of-view.
3 . The computer-implemented method of claim 1 , wherein determining global features representing relationships between the sets of local features comprises:
encoding the sets of local features with positional information; and
determining the global features based on the positional information.
4 . The computer-implemented method of claim 3 , wherein the positional information comprises pairwise positional relationships between the sets of local features.
5 . The computer-implemented method of claim 1 , wherein performing a medical imaging analysis task on the 3 D input medical image based on the hierarchical combination of the global features and the decoded sets of local features comprises:
performing a segmentation task based on the hierarchical combination of the global features and the decoded sets of local features.
6 . The computer-implemented method of claim 1 , wherein the medical imaging analysis task comprises one of classification, detection, or segmentation.
7 . The computer-implemented method of claim 1 , wherein the local feature extractor network is trained with self-supervised learning.
8 . The computer-implemented method of claim 1 , wherein the 3D input medical image depicts a brain of a patient.
9 . An apparatus comprising:
means for receiving a plurality of 3D (three dimensional) patches extracted from a 3D input medical image;
means for extracting a set of local features from each of the plurality of 3D patches using a machine learning based local feature extractor network;
means for reconstructing, using a machine learning based decoder network, the sets of local features output by a bottleneck layer of the machine learning based feature extractor network;
means for determining global features representing relationships between the sets of local features;
means for hierarchically combining the global features with the reconstructed sets of local features at a plurality of scales;
means for performing a medical imaging analysis task on the 3D input medical image based on the hierarchical combination of the global features and the decoded sets of local features; and
means for outputting results of the medical imaging analysis task.
10 . The apparatus of claim 9 , wherein the plurality of 3D patches is of a fixed resolution and a fixed field-of-view.
11 . The apparatus of claim 9 , wherein the means for determining global features representing relationships between the sets of local features comprises:
means for encoding the sets of local features with positional information; and
means for determining the global features based on the positional information.
12 . The apparatus of claim 11 , wherein the positional information comprises pairwise positional relationships between the sets of local features.
13 . A non-transitory computer readable medium storing computer program instructions, the computer program instructions when executed by a processor cause the processor to perform operations comprising:
receiving a plurality of 3D (three dimensional) patches extracted from a 3D input medical image;
extracting a set of local features from each of the plurality of 3D patches using a machine learning based local feature extractor network;
reconstructing, using a machine learning based decoder network, the sets of local features output by a bottleneck layer of the machine learning based feature extractor network;
determining global features representing relationships between the sets of local features;
hierarchically combining the global features with the reconstructed sets of local features at a plurality of scales;
performing a medical imaging analysis task on the 3D input medical image based on the hierarchical combination of the global features and the decoded sets of local features; and
outputting results of the medical imaging analysis task.
14 . The non-transitory computer readable medium of claim 13 , wherein the plurality of 3D patches is of a fixed resolution and a fixed field-of-view.
15 . The non-transitory computer readable medium of claim 13 , wherein performing a medical imaging analysis task on the 3D input medical image based on the hierarchical combination of the global features and the decoded sets of local features comprises:
performing a segmentation task based on the hierarchical combination of the global features and the decoded sets of local features.
16 . The non-transitory computer readable medium of claim 13 , wherein the medical imaging analysis task comprises one of classification, detection, or segmentation.
17 . The non-transitory computer readable medium of claim 13 , wherein the local feature extractor network is trained with self-supervised learning.
18 . The non-transitory computer readable medium of claim 13 , wherein the 3D input medical image depicts a brain of a patient.