Medical image analysis using neural networks
Systems and methods are described for automatically determining layer structure from medical image data. A processing device receives image data of biological layers captured by a medical imaging device. The processing device determines a boundary surface score for each pixel of the image data using a neural network, the boundary surface score being representative of a likelihood that each pixel corresponds to a boundary between segmented layers within the image data, to generate data defining boundary surfaces between segmented layers in the image data. In one embodiment, the neural network includes first and second sub-networks connected in series, the first sub-network configured with a multi-scale pooling layer that provides additional filters at respective defined sampling rates. The first sub-network processes the image data to generate segmentation data identifying a plurality of tissue layers in the input medical image, and the second sub-network processes the segmentation data to identify boundary surfaces between the plurality of tissue layers. Other embodiments are also described and claimed.
1 . A computer-implemented method for determining layer structure from medical image data, the method comprising:
receiving, by a processing device, image data of biological layers captured by a medical imaging device; and
determining, by the processing device, a boundary surface score for each pixel of the image data using a neural network, the boundary surface score being representative of a likelihood that each pixel corresponds to a boundary between segmented layers within the image data, to generate data defining boundary surfaces between segmented layers in the image data,
wherein the neural network includes first and second sub-networks connected in series, the first sub-network is configured with a multi-scale pooling layer that provides additional filters at respective defined sampling rates, and
wherein determining the boundary surface score includes:
using the first sub-network to process the image data to generate segmentation data identifying a plurality of tissue layers in the image data; and
using the second sub-network to process the segmentation data to identify boundary surfaces between the plurality of tissue layers.
2 . The method of claim 1 , further comprising generating an output image based on the image data and the identified boundary surfaces.
3 . The method of claim 1 , wherein the first sub-network comprises:
an encoder path having a plurality of down-sampling pooling layers and the multi-scale pooling layer; and
a decoder path that receives and processes output of the encoder path, the decoder path having a plurality of up-sampling pooling layers.
4 . The method of claim 3 , wherein the second sub-network comprises an encoder path having a plurality of down-sampling pooling layers that receives and processes output of the decoder path of the first sub-network.
5 . The method of claim 4 , wherein the encoder path of the second sub-network comprises a further multi-scale pooling layer.
6 . The method of claim 4 , wherein the second sub-network further comprises a decoder path that receives and processes output of the encoder path of the second sub-network, the decoder path of the second sub-network having a plurality of up-sampling pooling layers.
7 . The method of claim 1 , wherein the multi-scale pooling layer is an atrous spatial pyramid pooling layer including parallel filters with respective different scales defining a corresponding sampling rate.
8 . The method of claim 1 , further comprising training the first sub-network by determining updates to parameter values of the first sub-network based on a calculated difference between training segmentation maps and a segmentation map generated by the first sub-network.
9 . The method of claim 8 , wherein training the neural network further comprises determining updates to the parameter values of the first sub-network and initialised parameter values of the second sub-network together, based on a calculated difference between training boundary surface values and predicted boundary surface values generated by the second sub-network.
10 . The method of claim 1 , wherein training the neural network further comprises providing a third sub-network connected to the first sub-network, to output data identifying one or more additional features of the image data, wherein the first and third sub-networks are trained using back-propagation of a weighted combined loss calculated from the outputs of the third sub-network.
11 . The method of claim 10 , wherein training the neural network further comprises providing a fourth sub-network connected to the third sub-network, to output additional data identifying confounding features of the image data, wherein the first sub-network is trained using back-propagation of a weighted combined loss calculated from the outputs of the third sub-network with a penalty value derived from at least the outputs of the fourth sub-network.
12 . The method of claim 1 , further comprising generating a representation of reconstructed boundaries between the plurality of tissue layers using the determined boundary surface scores.
13 . The method of claim 1 , further comprising processing the determined boundary surface scores using a further trained classifier.
14 . The method of claim 1 , wherein the tissue layers are eye tissue layers captured by an optical coherence tomography device.
15 . The method of claim 1 , further comprising training the neural network by iterating through the following steps until optimised:
(i) train the first sub-network in isolation; and
(ii) train the second sub-network, using trained weights of the first sub-network.
16 . A method of training a neural network to determine layer structure from medical image data, the method comprising:
providing a first network to output data identifying one or more segmentation features in the medical image data, wherein the first network includes encoder and decoder sub-networks configured to receive and process the medical image data to generate a segmentation map;
providing a second network connected to the encoder sub-network of the first network, to output data identifying one or more additional pathological features of the medical image data;
wherein the first and second networks are trained using back-propagation of a weighted combined loss calculated from the outputs of at least the second network.
17 . The method of claim 16 , further comprising providing a third network to receive and process outputs from a multi-task network to generate additional data identifying confounding features in the medical image data.
18 . The method of claim 17 , wherein an adversarial network is connected in series to the multi-task network during training, and wherein weights of the multi-task network are optimised using a gradient of adversarial loss calculated with respect to at least the parameters in the multi-task network.
19 . A system configured to perform the method of claim 1 .
20 . A non-transitory storage medium comprising machine readable instructions stored thereon for causing a computer system to perform a method in accordance with claim 1 .