Protocol-aware tissue segmentation in medical imaging
For medical imaging such as MRI, machine training is used to train a network for segmentation using both the imaging data and protocol data (e.g., meta-data). The network is trained to segment based, in part, on the configuration and/or scanner, not just the imaging data, allowing the trained network to adapt to the way each image is acquired. In one embodiment, the network architecture includes one or more blocks that receive both types of data as input and output both types of data, preserving relevant features for adaptation through at least part of the trained network.
1. A method for segmentation by a magnetic resonance imager, the method comprising:
inputting both protocol and image data to a machine-learned network, the machine-learned network having been trained with layers, at least some of the layers each having input couples and output couples, both the input couples and the output couples being a coupling of and relating to the protocol and the image data;
outputting locations by the machine-learned network in response to the input of both the protocol and image data; and
displaying a magnetic resonance image showing the locations.
2. The method of claim 1 wherein the image data represents a brain of a patient, and wherein displaying comprises displaying the magnetic resonance image of the brain.
3. The method of claim 1 wherein at least one of the layers includes image features separated by type of tissue.
4. The method of claim 1 wherein each of the layers having both input and output couples includes normalization, linear, and non-linear portions, and wherein outputting comprises outputting based on the normalization, linear, and non-linear portions.
5. The method of claim 1 wherein inputting comprises inputting to the machine-learned network for different protocols used on different patients.
6. The method of claim 1 further comprising identifying the protocol data as a magnetic resonance weighting type or a feature derived from the magnetic resonance weighting type.
7. The method of claim 1 further comprising identifying a setting for a sequence parameter, geometrical information of a scan of a patient, and/or task specific information or a feature derived from the setting, geometrical information, and/or task specific information as the protocol data.
8. The method of claim 1 wherein inputting comprises inputting where the machine-learned network comprises a U-net including the one or more mixed blocks at a U-net layer other than an input or an output of the U-net.
9. The method of claim 1 wherein inputting comprises inputting where the machine-learned network comprises a U-net with a conditional network including a one or more mixed blocks, the conditional network outputting to a bottleneck of the U-net.
10. The method of claim 1 wherein inputting comprises inputting with the machine-learned network comprising a mixed block configured to receive first protocol information as the protocol data and first imaging information as the image data and to output second protocol information and second imaging information, the mixed block comprising:
a first normalization configured to output statistical information from the input imaging information to concatenate with the first protocol information and output normalized imaging information;
a first fully connected layer configured to output a scale value to invert the normalized imaging information;
a batch normalization receiving the concatenated first protocol information and the statistical information;
a second fully connected layer configured to receive an output of the batch normalization and output to a summer and a first non-linear activation function;
the first non-linear activation function configured to output the second protocol information;
a multiplier configured to invert the normalized imaging information based on the scale value;
a convolution layer configured to convolve with the inverted, normalized imaging information;
the summer configured to sum an output of the convolution layer with the output of the second fully connected layer; and
a second non-linear activation function configured to output the second imaging information in response to input of an output from the summer.
11. The method of claim 1 wherein inputting comprises inputting with the machine-learned network comprising an input layer having a mixed block configured to receive the protocol data and the imaging data and output second protocol information and second imaging imagining information.
12. The method of claim 1 wherein inputting comprises inputting with the machine-learned network including an instance normalization configured to output a skewness and/or kurtosis concatenated with the protocol information.
13. The method of claim 1 wherein the machine-learned network was trained as a multi-task training using uncertainty estimation.