Disease detection from weakly annotated volumetric medical images using convolutional long short-term memory
Systems and methods for developing a disease detection model. One method includes training the model using an image study and an associated disease label mined from a radiology report. The image study including a sequence of a plurality of two-dimensional slices of a three-dimensional image volume, and the model including a convolutional neural network layer and a convolutional long short-term memory layer. Training the model includes individually extracting a set of features from each of the plurality of two-dimensional slices using the convolutional neural network layer, sequentially processing the features extracted by the convolutional neural network layer for each of the plurality of two-dimensional slices using the convolutional long short-term memory layer, processing output from the convolutional long short-term memory layer for a sequentially last of the plurality of two-dimensional slices to generate a probability of the disease, and updating the model based on comparing the probability to the label.
1. A system for developing a disease detection model, the system comprising:
one or more electronic processors configured to:
train the disease detection model using an image study and an associated disease label mined from a radiology report generated for the image study, the image study including a sequence of a plurality of two-dimensional slices of a three-dimensional image volume, the disease detection model including a convolutional neural network layer and a convolutional long short-term memory layer, wherein training the disease detection model includes
individually extracting a feature of each of the plurality of two-dimensional slices using the convolutional neural network layer;
sequentially processing, as time series image data, the feature extracted by the convolutional neural network layer for each of the plurality of two-dimensional slices using the convolutional long short-term memory layer;
processing output from the convolutional long short-term memory layer for a sequentially last of the plurality of two-dimensional slices to generate a probability of the disease;
comparing the probability to the label;
updating the disease detection model based on the comparison; and
after the disease detection model is trained, apply the disease detection model to a new image study including a sequence of a plurality of two-dimensional slices of a three-dimensional image volume to generate a probability of the disease for the new image study.
2. The system of claim 1 , wherein the disease label includes a binary label of whether a patient associated with the image study was diagnosed with the disease.
3. The system of claim 1 , wherein the probability of the disease is a probability of emphysema.
4. The system of claim 1 , wherein the plurality of two-dimensional slices include a series of computed tomography (CT) images including a chest.
5. The system of claim 1 , wherein the three-dimensional image volume does not include any annotations.
6. The system of claim 1 , wherein sequentially processing the feature extracted by the convolution neural network layer for each of the plurality of two-dimensional slices using the convolution long short-term memory layer includes detecting spatial patterns and variations through the three-dimensional image volume.
7. The system of claim 1 , wherein the disease detection model includes two bi-directional pairs of units, wherein each unit includes the convolutional layer and the convolutional long short-term memory layer.
8. The system of claim 1 , wherein the two bi-directional pairs of units includes a first unit applying 32 filters to the three-dimensional image volume in an ascending order of the plurality of two-dimensional slices, a second unit applying 32 filters to the three-dimensional image volume in a descending order of the plurality of two-dimensional slices, a third unit applying 64 filters to the three-dimensional image volume in the ascending order, and a fourth unit applying 64 filters to the three-dimensional image volume in the descending order.
9. The system of claim 1 , wherein the one or more processors are configured to process the output from the convolutional long short-term memory layer for the sequentially last of the plurality of two-dimensional slices to generate a probability of the disease by outputting a single set of features from the sequentially last of the plurality of two-dimensional slices to a sigmoid-activated dense layer.
10. The system of claim 1 , wherein the disease detection model further includes a max pooling layer for the convolutional long short-term memory layer.
11. A method for developing a disease detection model, the method comprising:
training, with one or more electronic processors, the disease detection model using an image study and an associated disease label mined from a radiology report generated for the image study, the image study including a sequence of a plurality of two-dimensional slices of a three-dimensional image volume, the disease detection model including a convolutional neural network layer and a convolutional long short-term memory layer, wherein training the disease detection model includes
individually extracting a feature of each of the plurality of two-dimensional slices using the convolutional neural network layer;
sequentially processing, as time series image data, the feature extracted by the convolutional neural network layer for each of the plurality of two-dimensional slices using the convolutional long short-term memory layer;
processing output from the convolutional long short-term memory layer for a sequentially last of the plurality of two-dimensional slices to generate a probability of the disease;
comparing the probability to the label;
updating the disease detection model based on the comparison; and
after the disease detection model is trained, applying the disease detection model to a new image study including a sequence of a plurality of two-dimensional slices of a three-dimensional image volume to generate a probability of the disease for the new image study.
12. The method of claim 11 , wherein the disease label includes a binary label of whether a patient associated with the image study was diagnosed with the disease.
13. The method of claim 11 , wherein the probability of the disease is a probability of emphysema.
14. The method of claim 11 , wherein the plurality of two-dimensional slices include a series of computed tomography (CT) images including a chest.
15. The method of claim 11 , wherein the 3-dimensional image volume does not include any annotations.
16. The method of claim 11 , wherein the disease detection model includes two bi-directional pairs of units, wherein each unit includes the convolutional layer and the convolutional long short-term memory layer.
17. The method of claim 11 , wherein processing the output from the convolutional long short-term memory layer for the sequentially last of the plurality of two-dimensional slices to generate a probability of the disease includes outputting a single set of features from the sequentially last of the plurality of two-dimensional slices to a sigmoid-activated dense layer.
18. Non-transitory computer-readable medium storing instructions that, when executed by one or more electronic processors, perform a set of functions, the set of functions comprising:
training the disease detection model using an image study and an associated disease label mined from a radiology report generated for the image study, the disease label including a binary label of whether a patient associated with the image study was diagnosed with the disease and the image study including a sequence of a plurality of two-dimensional slices of a three-dimensional image volume, the disease detection model including a convolutional neural network layer and a convolutional long short-term memory layer, wherein training the disease detection model includes
individually extracting a feature of each of the plurality of two-dimensional slices using the convolutional neural network layer;
sequentially processing, as time series image data, the feature extracted by the convolutional neural network layer for each of the plurality of two-dimensional slices using the convolutional long short-term memory layer;
processing output from the convolutional long short-term memory layer for a sequentially last of the plurality of two-dimensional slices to generate a probability of the disease;
comparing the probability to the label;
updating the disease detection model based on the comparison; and
after the disease detection model is trained, applying the disease detection model to a new image study including a sequence of a plurality of two-dimensional slices of a three-dimensional image volume to generate a probability of the disease for the new image study.
19. The non-transitory computer-readable medium of claim 18 , wherein the probability of the disease is a probability of emphysema and wherein the plurality of two-dimensional slices include a series of computed tomography (CT) images including a chest.
20. The non-transitory computer-readable medium of claim 18 , wherein the three-dimensional image volume does not include any annotations.