Protocol-aware tissue segmentation in medical imaging
For medical imaging such as MRI, machine training is used to train a network for segmentation using both the imaging data and protocol data (e.g., meta-data). The network is trained to segment based, in part, on the configuration and/or scanner, not just the imaging data, allowing the trained network to adapt to the way each image is acquired. In one embodiment, the network architecture includes one or more blocks that receive both types of data as input and output both types of data, preserving relevant features for adaptation through at least part of the trained network.
1. A method for segmentation by a medical imager, the method comprising:
identifying protocol data for a protocol used to acquire imaging data representing a patient;
segmenting the imaging data with a machine-learned network, the machine-learned network outputting a segmentation in response to input of the protocol data and the imaging data, the machine-learned network comprising one or more mixed blocks with learned parameters, the one or more mixed blocks each configured to receive first protocol information and first imaging information and output second protocol information and second imaging imagining information; and
displaying an image representing the segmentation.
2. The method of claim 1 wherein identifying the protocol data comprises identifying a magnetic resonance weighting type, and wherein the protocol information comprises the protocol data or a feature derived from the protocol data.
3. The method of claim 1 wherein identifying the protocol data comprises identifying a setting for a sequence parameter, geometrical information of a scan of the patient, and/or task specific information, and wherein the protocol information comprises the protocol data or a feature derived from the protocol data.
4. The method of claim 1 wherein segmenting comprises segmenting with the machine-learned network, the machine-learned network comprising a U-net including the one or more mixed blocks at a layer other than an input or an output of the U-net.
5. The method of claim 1 wherein segmenting comprises segmenting with the machine-learned network, the machine-learned network comprising a U-net with a conditional network including the one or more mixed blocks, the conditional network outputting to a bottleneck of the U-net.
6. The method of claim 1 wherein segmenting comprises segmenting with the machine-learned network, the mixed block comprising a normalization layer, a linear layer, and a non-linear layer.
7. The method of claim 1 wherein segmenting comprises segmenting with the machine-learned network, the mixed block comprising:
a first normalization configured to output statistical information from the input imaging information to concatenate with the first protocol information and output normalized imaging information;
a first fully connected layer configured to output a scale value to invert the normalized imaging information;
a batch normalization receiving the concatenated first protocol information and the statistical information;
a second fully connected layer configured to receive an output of the batch normalization and output to a summer and a first non-linear activation function;
the first non-linear activation function configured to output the second protocol information;
a multiplier configured to invert the normalized imaging information based on the scale value;
a convolution layer configured to convolve with the inverted, normalized imaging information;
the summer configured to sum an output of the convolution layer with the output of the second fully connected layer; and
a second non-linear activation function configured to output the second imaging information in response to input of an output from the summer.
8. The method of claim 1 wherein segmenting comprises segmenting with the machine-learned network, the machine-learned network comprising an input layer having the mixed block, the first protocol information comprising the protocol data, and the first imaging information comprising the imaging data.
9. The method of claim 1 wherein segmenting comprises segmenting with the machine-learned network, the mixed block including an instance normalization configured to output a skewness and/or kurtosis concatenated with the protocol information.
10. The method of claim 1 wherein segmenting comprises segmenting with the machine-learned network, the mixed block configured to receive the first protocol information as multiple protocol features output by a previous layer of the machine-learned network, and the mixed block configured to receive the first imaging data as multiple imaging features output by the previous layer of the machine-learned network.
11. The method of claim 1 wherein the machine-learned network was trained as a multi-task training using uncertainty estimation.