Disease detection from weakly annotated volumetric medical images using convolutional long short-term memory and multiple instance learning

1. A system for developing a disease detection model, the system comprising:

one or more electronic processors configured to:

train the disease detection model using an image study and an associated disease label mined from a radiology report generated for the image study, the image study including a sequence of a plurality of two-dimensional slices of a three-dimensional image volume, the model including a convolutional neural network layer and a convolutional long short-term memory layer, wherein training the disease detection model includes

individually extracting a set of features from each of the plurality of two-dimensional slices using the convolutional neural network layer;

sequentially processing the set of features extracted by the convolutional neural network layer for each of the plurality of two-dimensional slices using the convolutional long short-term memory layer;

processing output from the convolutional long short-term memory layer representing a probability of the disease for each of the plurality of two-dimensional slices to generate, using a multiple instance learning approach, an overall probability of the disease for the image study;

comparing the overall probability to the disease label;

updating the disease detection model based on the comparison; and

after the disease detection model is trained, apply the disease detection model to a new image study to generate a probability of the disease for the new image study,

wherein applying the disease detection model to the new image study includes processing the new image study with two bi-directional pairs of units, wherein each unit includes the convolutional layer and the convolutional long short-term memory layer, wherein the two bi-directional pairs of units includes a first unit, a second unit, a third unit, and a fourth unit, the first unit including a first convolutional layer and a first convolutional long short-term memory layer, the second unit including a second convolutional layer and a second convolutional long short-term memory layer, the third unit including a third convolutional layer and a third convolutional long short-term memory layer, and the fourth unit including a fourth convolutional layer and a fourth convolutional long short-term memory layer, and wherein sequentially processing the set of features includes processing, with the first unit of the two bi-directional pairs of units, the plurality of two-dimensional slices with the first convolutional layer and the first convolutional long short-term memory layer in a first order and processing, with the second unit of the two bi-directional pairs of units, the plurality of two-dimensional slices with the second convolutional layer and the second convolutional short-term memory layer in a second order, the second order opposite to the first order.

2. The system of claim 1 , wherein processing the probability for each of the plurality of two-dimensional slices includes performing mean pooling.

3. The system of claim 1 , wherein processing the probability for each of the plurality of two-dimensional slices includes performing max pooling.

4. The system of claim 1 , wherein processing the probability for each of the plurality of two-dimensional slices includes performing product pooling.

5. The system of claim 1 , wherein the disease label includes a binary label of whether a patient associated with the image study was diagnosed with the disease.

6. The system of claim 1 , wherein the overall probability of the disease is a probability of emphysema.

7. The system of claim 1 , wherein the plurality of two-dimensional slices include a series of computed tomography (CT) images including a chest.

8. The system of claim 1 , wherein the three-dimensional image volume does not include any annotations.

9. The system of claim 1 , wherein sequentially processing the set of features extracted by the convolution neural network layer for each of the plurality of two-dimensional slices using the convolution long short-term memory layer includes detecting spatial patterns and variations through the three-dimensional image volume.

10. The system of claim 1 , wherein the first unit applies 32 filters to the three-dimensional image volume in an ascending order of the plurality of two-dimensional slices, the second unit applies 32 filters to the three-dimensional image volume in a descending order of the plurality of two-dimensional slices, the third unit applies 64 filters to the three-dimensional image volume in the ascending order, and the fourth unit applies 64 filters to the three-dimensional image volume in the descending order.

11. The system of claim 1 , wherein the disease detection model further includes a max pooling layer for the convolutional long short-term memory layer.

12. A method for developing a disease detection model, the method comprising:

training, with one or more electronic processors, the disease detection model using an image study and an associated disease label mined from a radiology report generated for the image study, the image study including a sequence of a plurality of two-dimensional slices of a three-dimensional image volume, the disease detection model including a convolutional neural network layer and a convolutional long short-term memory layer, wherein training the disease detection model includes

individually extracting a set of features from each of the plurality of two-dimensional slices using the convolutional neural network layer;

sequentially processing the set of features extracted by the convolutional neural network layer for each of the plurality of two-dimensional slices using the convolutional long short-term memory layer;

processing output from the convolutional long short-term memory layer representing a probability of the disease for each of the plurality of two-dimensional slices to generate, using a multiple learning approach, an overall probability of the disease for the image study;

comparing the overall probability to the disease label;

updating the disease detection model based on the comparison; and

after the disease detection model is trained, applying the disease detection model to a new image study to generate a probability of the disease for the new image study

wherein applying the disease detection model to the new image study includes processing the new image study with two bi-directional pairs of units, wherein each unit includes the convolutional layer and the convolutional long short-term memory layer, wherein the two bi-directional pairs of units includes a first unit, a second unit, a third unit, and a fourth unit, the first unit including a first convolutional layer and a first convolutional long short-term memory layer, the second unit including a second convolutional layer and a second convolutional long short-term memory layer, the third unit including a third convolutional layer and a third convolutional long short-term memory layer, and the fourth unit including a fourth convolutional layer and a fourth convolutional long short-term memory layer, and wherein sequentially processing the set of features includes processing, with the first unit of the two bi-directional pairs of units, the plurality of two-dimensional slices with the first convolutional layer and the first convolutional long short-term memory layer in a first order and processing, with the second unit of the two bi-directional pairs of units, the plurality of two-dimensional slices with the second convolutional layer and the second convolutional short-term memory layer in a second order, the second order opposite to the first order.

13. The method of claim 12 , wherein processing the probability for each of the plurality of two-dimensional slices includes performing mean pooling.

14. The method of claim 12 , wherein processing the probability for each of the plurality of two-dimensional slices includes performing max pooling.

15. The method of claim 12 , wherein processing the probability for each of the plurality of two-dimensional slices includes performing product pooling.

16. Non-transitory computer-readable medium storing instructions that, when executed by one or more electronic processors, perform a set of functions, the set of functions comprising:

training the disease detection model using an image study and an associated disease label mined from a radiology report generated for the image study, the disease label including a binary label of whether a patient associated with the image study was diagnosed with the disease and the image study including a sequence of a plurality of two-dimensional slices of a three-dimensional image volume, the disease detection model including a convolutional neural network layer and a convolutional long short-term memory layer, wherein training the disease detection model includes

individually extracting a set of features from each of the plurality of two-dimensional slices using the convolutional neural network layer;

sequentially processing the set of features extracted by the convolutional neural network layer for each of the plurality of two-dimensional slices using the convolutional long short-term memory layer;

processing output from the convolutional long short-term memory layer representing a probability of the disease for each of the plurality of two-dimensional slices to generate, using a multiple instance learning approach, an overall probability of the disease for the image study;

comparing the overall probability to the disease label;

updating the disease detection model based on the comparison; and

after the disease detection model is trained, applying the disease detection model to a new image study to generate a probability of the disease for the new image study

wherein applying the disease detection model to the new image study includes processing the new image study with two bi-directional pairs of units, wherein each unit includes the convolutional layer and the convolutional long short-term memory layer, wherein the two bi-directional pairs of units includes a first unit, a second unit, a third unit, and a fourth unit, the first unit including a first convolutional layer and a first convolutional long short-term memory layer, the second unit including a second convolutional layer and a second convolutional long short-term memory layer, the third unit including a third convolutional layer and a third convolutional long short-term memory layer, and the fourth unit including a fourth convolutional layer and a fourth convolutional long short-term memory layer, and wherein sequentially processing the set of features includes processing, with the first unit of the two bi-directional pairs of units, the plurality of two-dimensional slices with the first convolutional layer and the first convolutional long short-term memory layer in a first order and processing, with the second unit of the two bi-directional pairs of units, the plurality of two-dimensional slices with the second convolutional layer and the second convolutional short-term memory layer in a second order, the second order opposite to the first order.

17. The non-transitory computer-readable medium of claim 16 , wherein processing the probability for each of the plurality of two-dimensional slices includes performing one selected from a group consisting of mean pooling, max pooling, and product pooling.