Systems, methods, and media for automatically diagnosing intraductal papillary mucinous neosplasms using multi-modal magnetic resonance imaging data

1. A system for automatically detecting the presence of intraductal papillary mucinous neoplasms (IPMNs) in a subject's pancreas, the system comprising:

at least one hardware processor that is programmed to:

receive T1 MRI data generated by an MRI machine, the T1 MRI data comprising a plurality of slices of T1 MRI data [I 1 , . . . , I u , . . . , I N 1 ];

receive T2 MRI data generated by the MRI machine, the T2 MRI data comprising a plurality of slices of T2 MRI data [J 1 , . . . , J v , . . . , J N 2 ]

provide data representing k slices of the T1 MRI data to a trained image classification convolutional neural network (CNN);

provide data representing k slices of the T2 MRI data to the trained image classification CNN;

receive output from the trained image classification CNN;

determine, based on the output, that IPMNs are likely present in the subject's pancreas;

in response to determining that IPMNs are likely present in the subject's pancreas, cause an indication that IPMNs are likely present in the subject's pancreas to be presented to the user.

2. The system of claim 1 , further comprising the MRI scanner.

3. The system of claim 1 , wherein the at least one hardware processor that is further programmed to:

present the T1 MRI data and prompt a user to select a slice of T1 MRI data including the subject's pancreas; and

receive a selection of slice I u the T1 MRI data.

4. The system of claim 1 , wherein the at least one hardware processor is further programmed to:

generate a minimum intensity projection based on consecutive slices [I u−m , . . . , I u , . . . , I u+m ] where m is greater than or equal to 2, and the data representing the k slices of the T1 MRI data comprises the minimum intensity projection;

generate a maximum intensity projection based on consecutive slices [J v−m , . . . , J v , . . . , J v+m ] where m is greater than or equal to 2, and the data representing the k slices of the T2 MRI data comprises the maximum intensity projection;

provide the minimum intensity projection to the image classification CNN;

receive, from a fully connected layer of the image classification CNN, the output comprising a set of features Φ generated by the image classification CNN based on the minimum intensity projection;

provide the maximum intensity projection to the CNN;

receive, from the fully connected layer of the image classification CNN, a set of features Ψ generated by the image classification CNN based on the maximum intensity projection;

calculate a feature matrix based on a canonical correlation analysis (CCA) between features Φ and features Ψ;

provide the feature matrix as input to a support vector machine (SVM) trained to automatically detect the presence of IPMNs in multi-modal MRI data corresponding to an imaged pancreas based on an input feature matrix generated from T1 and T2 MRI data corresponding to the imaged pancreas;

receive an output from the SVM that is indicative of the presence of IPMNs in slice I u and slice J v ; and

determine, based on the output from the SVM, that IPMNs are likely present in the subject's pancreas.

5. The system of claim 1 , wherein the at least one hardware processor is further programmed to:

provide the k slices of the T1 MRI data to the trained image classification CNN, wherein the trained image classification CNN was generated using transfer learning to retrain at least one layer of a 3D image classification CNN that was expanded from a pre-trained general image classification CNN having a 2D architecture, the data representing the k slices of the T1 MRI data comprising the k slices of the T1 MRI data, and the weights of a plurality of kernels of the trained classification CNN are divided at least by a number of depth layers d added to the 2D architecture that is equal to k;

provide the k slices of the T2 MRI data to the trained image classification CNN; and

receive the output from the trained image classification CNN.

6. The system of claim 5 , wherein the at least one hardware processor is further programmed to:

concatenate the T1 MRI data and the T2 MRI data prior to providing the T2 MRI data and the T2 MRI data to the trained image classification CNN.

7. The system of claim 5 , wherein the at least one hardware processor is further programmed to:

provide the k slices of the T1 MRI data to a first convolutional layer of the trained image classification CNN;

provide the k slices of the T2 MRI data to a second convolutional layer of the trained image classification CNN;

concatenate an output of the first convolutional layer and an output of the second convolution layer; and

provide a set of features based on the concatenated outputs to a third convolutional layer.

8. A method for automatically detecting the presence of intraductal papillary mucinous neoplasms (IPMNs) in a subject's pancreas, the method comprising:

receiving T1 MRI data generated by an MRI machine, the T1 MRI data comprising a plurality of slices of T1 MRI data [I 1 , . . . , I u , . . . , I N 1 ];

receiving T2 MRI data generated by the MRI machine, the T2 MRI data comprising a plurality of slices of T2 MRI data [J 1 , . . . , J v , . . . , J N 2 ];

providing data representing k slices of the T1 MRI data to a trained image classification convolutional neural network (CNN);

providing data representing k slices of the T2 MRI data to the trained image classification CNN;

receiving output from the trained image classification CNN;

determining, based on the output, that IPMNs are likely present in the subject's pancreas; and

in response to determining that IPMNs are likely present in the subject's pancreas, causing an indication that IPMNs are likely present in the subject's pancreas to be presented to the user.

9. The method of claim 8 , further comprising:

presenting the T1 MRI data and prompt a user to select a slice of T1 MRI data including the subject's pancreas; and

receiving a selection of slice I u of the T1 MRI data.

10. The method of claim 8 , further comprising:

generating a minimum intensity projection based on consecutive slices [I u−m , . . . , I u , . . . , I u+m ] where m is greater than or equal to 2, and the data representing the k slices of the T1 MRI data comprises the minimum intensity projection;

generating a maximum intensity projection based on consecutive slices [J v−m , . . . , J v , . . . , J v+m ] where m is greater than or equal to 2, and the data representing the k slices of the T2 MRI data comprises the maximum intensity projection;

providing the minimum intensity projection to the image classification CNN;

receiving, from a fully connected layer of the image classification CNN, the output comprising a set of features Φ generated by the image classification CNN based on the minimum intensity projection;

providing the maximum intensity projection to the CNN;

receiving, from the fully connected layer of the image classification CNN, a set of features Ψ generated by the image classification CNN based on the maximum intensity projection;

calculating a feature matrix based on a canonical correlation analysis (CCA) between features Φ and features Ψ;

providing the feature matrix as input to a support vector machine (SVM) trained to automatically detect the presence of IPMNs in multi-modal MRI data corresponding to an imaged pancreas based on an input feature matrix generated from T1 and T2 MRI data corresponding to the imaged pancreas;

receiving an output from the SVM that is indicative of the presence of IPMNs in slice I u and slice J v ; and

determining, based on the output from the SVM, that IPMNs are likely present in the subject's pancreas.

11. The method of claim 8 , further comprising:

providing the k slices of the T1 MRI data to the trained image classification CNN, wherein the trained image classification CNN was generated using transfer learning to retrain at least one layer of a 3D image classification CNN that was expanded from a pre-trained general image classification CNN having a 2D architecture, the data representing the k slices of the T1 MRI data comprising the k slices of the T1 MRI data, and the weights of a plurality of kernels of the trained classification CNN are divided at least by a number of depth layers d added to the 2D architecture that is equal to k;

providing the k slices of the T2 MRI data to the trained image classification CNN; and

receiving the output from the trained image classification CNN.

12. The method of claim 11 , further comprising:

concatenating the T1 MRI data and the T2 MRI data prior to providing the T1 MRI data and the T2 MRI data to the trained image classification CNN.

13. The method of claim 11 , further comprising:

providing the k slices of the T1 MRI data to a first convolutional layer of the trained image classification CNN;

provide the k slices of the T2 MRI data to a second convolutional layer of the trained image classification CNN;

concatenate an output of the first convolutional layer and an output of the second convolution layer; and

provide a set of features based on the concatenated outputs to a third convolutional layer.

14. A non-transitory computer readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for automatically detecting the presence of intraductal papillary mucinous neoplasms (IPMNs) in a subject's pancreas, the method comprising:

receiving T1 MRI data generated by an MRI machine, the T1 MRI data comprising a plurality of slices of T1 MRI data [I 1 , . . . , I u , . . . , I N 1 ];

receiving T2 MRI data generated by the MRI machine, the T2 MRI data comprising a plurality of slices of T2 MRI data [J 1 , . . . , J v , . . . , J N 2 ];

providing data representing k slices of the T1 MRI data to a trained image classification convolutional neural network (CNN);

providing data representing k slices of the T2 MRI data to the trained image classification CNN;

receiving output from the trained image classification CNN;

determining, based on the output, that IPMNs are likely present in the subject's pancreas; and

in response to determining that IPMNs are likely present in the subject's pancreas, causing an indication that IPMNs are likely present in the subject's pancreas to be presented to the user.

15. The non-transitory computer readable medium of claim 14 , wherein the method further comprises:

presenting the T1 MRI data and prompt a user to select a slice of T1 MRI data including the subject's pancreas; and

receiving a selection of slice I u of the T1 MRI data.

16. The non-transitory computer readable medium of claim 14 , wherein the method further comprises:

generating a minimum intensity projection based on consecutive slices [I u−m , . . . , I u , . . . , I u+m ] where m is greater than or equal to 2, and the data representing the k slices of the T1 MRI data comprises the minimum intensity projection;

generating a maximum intensity projection based on consecutive slices [J v−m , . . . , J v , . . . , J v+m ] where m is greater than or equal to 2, and the data representing the k slices of the T2 MRI data comprises the maximum intensity projection;

providing the minimum intensity projection to the image classification CNN;

receiving, from a fully connected layer of the image classification CNN, the output comprising a set of features Φ generated by the image classification CNN based on the minimum intensity projection;

providing the maximum intensity projection to the CNN;

receiving, from the fully connected layer of the image classification CNN, a set of features Ψ generated by the image classification CNN based on the maximum intensity projection;

calculating a feature matrix based on a canonical correlation analysis (CCA) between features Φ and features Ψ;

providing the feature matrix as input to a support vector machine (SVM) trained to automatically detect the presence of IPMNs in multi-modal MRI data corresponding to an imaged pancreas based on an input feature matrix generated from T1 and T2 MRI data corresponding to the imaged pancreas;

receiving an output from the SVM that is indicative of the presence of IPMNs in slice I u and slice J v ; and

determining, based on the output from the SVM, that IPMNs are likely present in the subject's pancreas.

17. The non-transitory computer readable medium of claim 14 , wherein the method further comprises:

providing the k slices of the T1 MRI data to the trained image classification CNN, wherein the trained image classification CNN was generated using transfer learning to retrain at least one layer of a 3D image classification CNN that was expanded from a pre-trained general image classification CNN having a 2D architecture, the data representing the k slices of the T1 MRI data comprising the k slices of the T1 MRI data, and the weights of a plurality of kernels of the trained classification CNN are divided at least by a number of depth layers d added to the 2D architecture that is equal to k;

providing the k slices of the T2 MRI data to the trained image classification CNN; and

receiving the output from the trained image classification CNN.

18. The non-transitory computer readable medium of claim 17 , wherein the method further comprises:

concatenating the T1 MRI data and the T2 MRI data prior to providing the T1 MRI data and the T2 MRI data to the trained image classification CNN.

19. The non-transitory computer readable medium of claim 17 , wherein the method further comprises:

providing the k slices of the T1 MRI data to a first convolutional layer of the trained image classification CNN;

provide the k slices of the T2 MRI data to a second convolutional layer of the trained image classification CNN;

concatenate an output of the first convolutional layer and an output of the second convolution layer; and

provide a set of features based on the concatenated outputs to a third convolutional layer.