Protocol-aware tissue segmentation in medical imaging
For medical imaging such as MRI, machine training is used to train a network for segmentation using both the imaging data and protocol data (e.g., meta-data). The network is trained to segment based, in part, on the configuration and/or scanner, not just the imaging data, allowing the trained network to adapt to the way each image is acquired. In one embodiment, the network architecture includes one or more blocks that receive both types of data as input and output both types of data, preserving relevant features for adaptation through at least part of the trained network.
1. A system for segmentation of brain tissue in magnetic resonance imaging (MRI), the system comprising:
an MRI scanner configured to scan the brain tissue of a patient, the scan providing scan data and meta data, the meta data representing the scan and the scan data representing the patient;
an image processor configured to segment between different types of the brain tissue by application of the scan data and the meta data to a machine-learned model, the machine-learned model outputting the segmentation, wherein the machine-learned network comprises an input layer having a mixed block configured to receive the meta data and the scan data and output second meta information and second scan information;
a display configured to display the segmentation of the brain tissue.
2. The system of claim 1 wherein the machine-learned model comprises a neural network with a first block having input channels for first features related to the meta data and for first features related to the scan data and having output channels for second features related to the meta data and for second features related to the scan data.
3. The system of claim 2 wherein the first block is configured to pass information from the first features related to the meta data to a path from the input of the first features related to the scan data to the output of the second features related to the scan data.
4. The system of claim 2 wherein the first block includes normalization, linear, and non-linearity layers.
5. The system of claim 1 wherein the scan data comprises image data and wherein the meta data comprises protocol data.
6. The system of claim 1 wherein the machine-learned network is configured to output the segmentation in response to input of both the scan data and the meta data.
7. The system of claim 1 wherein the machine-learned model is a neural network comprising layers including the input layer, wherein at least one of the layers includes image features separated by type of tissue.
8. The system of claim 1 wherein the machine-learned model is configured to receive input for different protocols used on different patients.
9. The system of claim 1 wherein the meta data is a magnetic resonance weighting type or a feature derived from the magnetic resonance weighting type.
10. The system of claim 1 wherein the meta data is a setting for a sequence parameter, geometrical information of a scan of the patient, task specific information and/or a feature derived from the setting, geometrical information, and/or task specific information as the protocol data.
11. The system of claim 1 wherein the machine-learned network comprises a U-net including the one or more additional mixed blocks at a U-net layer other than the input layer or an output layer of the U-net.
12. The system of claim 1 wherein the machine-learned network comprises a U-net with a conditional network including one or more additional mixed blocks, the conditional network outputting to a bottleneck of the U-net.
13. The system of claim 1 wherein the machine-learned network includes an instance normalization configured to output a skewness and/or kurtosis concatenated with the meta data.
14. The system of claim 1 wherein the machine-learned network was trained as a multi-task training using uncertainty estimation.
15. A system for segmentation of brain tissue in magnetic resonance imaging (MRI), the system comprising:
an MRI scanner configured to scan the brain tissue of a patient, the scan providing scan data and meta data, the meta data representing the scan and the scan data representing the patient;
an image processor configured to segment between different types of the brain tissue by application of the scan data and the meta data to a machine-learned model, the machine-learned model outputting the segmentation;
a display configured to display the segmentation of the brain tissue;
wherein the machine-learned network was trained with layers, one of the layers including the block and at least some of the other layers each having input couples and output couples, both the input couples and the output couples being a coupling of and relating to the scan data and the meta data.
16. A system for segmentation of brain tissue in magnetic resonance imaging (MRI), the system comprising:
an MRI scanner configured to scan the brain tissue of a patient, the scan providing scan data and meta data, the meta data representing the scan and the scan data representing the patient;
an image processor configured to segment between different types of the brain tissue by application of the scan data and the meta data to a machine-learned model, the machine-learned model outputting the segmentation;
a display configured to display the segmentation of the brain tissue;
wherein the machine-learned network comprises a mixed block configured to receive first protocol information as the meta data and first imaging information as the scan data and to output second protocol information and second imaging information, the mixed block comprising:
a first normalization configured to output statistical information from the input imaging information to concatenate with the first protocol information and output normalized imaging information;
a first fully connected layer configured to output a scale value to invert the normalized imaging information;
a batch normalization receiving the concatenated first protocol information and the statistical information;
a second fully connected layer configured to receive an output of the batch normalization and output to a summer and a first non-linear activation function;
the first non-linear activation function configured to output the second protocol information;
a multiplier configured to invert the normalized imaging information based on the scale value;
a convolution layer configured to convolve with the inverted, normalized imaging information;
the summer configured to sum an output of the convolution layer with the output of the second fully connected layer; and
a second non-linear activation function configured to output the second imaging information in response to input of an output from the summer.