Multimodal system and method for predicting cancer

1 . A method for developing a set of machine learning models for generating a cancer prediction for one or more prospective patients, comprising:

obtaining a batch of images captured from one or more patients, wherein obtaining the batch of images comprises obtaining a batch of microscopic training images comprising tissue structures and obtaining a batch of macroscopic training images comprising radiomic features;

training a first convolutional neural network model by providing, as input, the batch of microscopic training images until an improvement in first output data between each epoch of training is less than a first pre-defined threshold;

training a second convolutional neural network model by providing, as input, the batch of macroscopic training images until an improvement in second output data between each epoch of training is less than a second pre-defined threshold;

training a first machine learning model by providing, as input, extracted features from the batch of microscopic training images until an improvement in third output data between each epoch of training is less than a third pre-defined threshold;

training a second machine learning model by providing, as input, extracted features from the batch of macroscopic training images until an improvement in fourth output data between each epoch of training is less than a fourth pre-defined threshold;

locking the first and second convolutional neural network models and the first and second machine learning models;

generating a first set of output features from each epoch of training in the first convolutional neural network model and the first machine learning model;

generating a second set of output features from each epoch of training in the second convolutional neural network model and the second machine learning model;

obtaining a batch of electronic health record (EHR) data; and

training a fully connected network by using the first set of output features, the second set of output features, and the batch of EHR data until an improvement in fifth output data between each epoch of training is less than a fifth pre-defined threshold.

2 . The method of claim 1 , wherein training the fully connected network using the first set of output features, the second set of output features, and the batch of EHR data comprises:

concatenating the first set of output features, the second set of output features, and the batch of EHR data into a feature vector; and

training the fully connected network on the feature vector.

3 . The method of claim 1 ,

wherein training the first machine learning model comprises:

extracting first image features from the microscopic training images; and

training the first machine learning model on the first image features, and

wherein training the second machine learning model comprises:

extracting second image features from the macroscopic training images; and

training the second machine learning model on the second image features.

4 . The method of claim 1 ,

wherein training the first convolutional neural network model comprises:

extracting first image features from the microscopic training images; and

training the first convolutional neural network model on the first image features, and

wherein training the second convolutional neural network model comprises:

extracting second image features from the macroscopic training images; and

training the second convolutional neural network model on the second image features.

5 . The method of claim 1 , wherein the first convolutional neural network model and the second convolutional neural network model each comprises any one of a VGGNet model, a ResNet model, and an inception model.

6 . The method of claim 1 , wherein the first machine learning model and the second machine learning model each comprise any one of a logistic regression model, a cox proportional-hazards model, and a random survival forest model.

7 . The method of claim 1 , wherein the macroscopic training images comprise radiology images, and wherein the radiology images capture at least one of: one or more tumor areas, surrounding areas in proximity to the one or more tumor areas, or lymph nodes.

8 . The method of claim 1 , wherein the microscopic training images comprise histology images, and wherein the histology images capture at least one of: one or more cancer cells, lymphocytes, stromal cells, or epithelial cells.

9 . The method of claim 1 , wherein the fully connected network comprises any one of a neural network, a random forest, or a support vector machine.

10 . The method of claim 1 , wherein the fully connected network is trained to generate outputs informative of at least one of: the cancer prediction as a composite risk index, one or more treatment decisions, or clinical trial recruitment predictions.

11 . A non-transitory computer-readable storage medium storing instruction for developing a set of models for generating a cancer prediction for one or more prospective patients, the instructions when executed by a processor causing the processor to perform steps comprising:

obtaining a batch of images captured from one or more patients, wherein obtaining the batch of images comprises obtaining a batch of microscopic training images comprising tissue structures and obtaining a batch of macroscopic training images comprising radiomic features;

training a first convolutional neural network model by providing, as input, the batch of microscopic training images until an improvement in first output data between each epoch of training is less than a first pre-defined threshold;

training a second convolutional neural network model by providing, as input, the batch of macroscopic training images until an improvement in second output data between each epoch of training is less than a second pre-defined threshold;

training a first machine learning model by providing, as input, extracted features from the batch of microscopic training images until an improvement in third output data between each epoch of training is less than a third pre-defined threshold;

training a second machine learning model by providing, as input, extracted features from the batch of macroscopic training images until an improvement in fourth output data between each epoch of training is less than a fourth pre-defined threshold;

locking the first and second convolutional neural network models and the first and second machine learning models;

generating a first set of output features from each epoch of training in the first convolutional neural network model and the first machine learning model;

generating a second set of output features from each epoch of training in the second convolutional neural network model and the second machine learning model;

obtaining a batch of electronic health record (EHR) data; and

training a fully connected network by using the first set of output features, the second set of output features, and the batch of EHR data until an improvement in fifth output data between each epoch of training is less than a fifth pre-defined threshold.

12 . The non-transitory computer-readable storage medium of claim 11 , wherein training the fully connected network using the first set of output features, the second set of output features, and the batch of EHR data comprises:

concatenating the first set of output features, the second set of output features, and the batch of EHR data into a feature vector; and

training the fully connected network on the feature vector.

13 . The non-transitory computer-readable storage medium of claim 11 ,

wherein training the first machine learning model comprises:

extracting first image features from the microscopic training images; and

training the first machine learning model on the first image features, and

wherein training the second machine learning model comprises:

extracting second image features from the macroscopic training images; and

training the second machine learning model on the second image features.

14 . The non-transitory computer-readable storage medium of claim 11 ,

wherein training the first convolutional neural network model comprises:

extracting first image features from the microscopic training images; and

training the first convolutional neural network model on the first image features, and

wherein training the second convolutional neural network model comprises:

extracting second image features from the macroscopic training images; and

training the second convolutional neural network model on the second image features.

15 . The non-transitory computer-readable storage medium of claim 11 , wherein the first convolutional neural network model and the second convolutional neural network model each comprises any one of a VGGNet model, a ResNet model, and an inception model.

16 . The non-transitory computer-readable storage medium of claim 11 , wherein the first machine learning model and the second machine learning model each comprise any one of a logistic regression model, a cox proportional-hazards model, and a random survival forest model.

17 . The non-transitory computer-readable storage medium of claim 11 , wherein the microscopic training images comprise histology images, and wherein the histology images capture at least one of: one or more cancer cells, lymphocytes, stromal cells, or epithelial cells.

18 . The non-transitory computer-readable storage medium of claim 11 , wherein the macroscopic training images comprise radiology images, and wherein the radiology images capture at least one of: one or more tumor areas, surrounding areas in proximity to the one or more tumor areas, or lymph nodes.

19 . The non-transitory computer-readable storage medium of claim 11 , wherein the fully connected network comprises any one of a neural network, a random forest, or a support vector machine.

20 . A computer system comprising:

a processor; and

a non-transitory computer-readable storage medium storing instruction for developing a set of models for generating a cancer prediction for one or more prospective patients, the instructions when executed by the processor causing the processor to perform steps comprising:

obtaining a batch of images captured from one or more patients, wherein obtaining the batch of images comprises obtaining a batch of microscopic training images comprising tissue structures and obtaining a batch of macroscopic training images comprising radiomic features;

training a first convolutional neural network model by providing, as input, the batch of microscopic training images until an improvement in first output data between each epoch of training is less than a first pre-defined threshold;

training a second convolutional neural network model by providing, as input, the batch of macroscopic training images until an improvement in second output data between each epoch of training is less than a second pre-defined threshold;

training a first machine learning model by providing, as input, extracted features from the batch of microscopic training images until an improvement in third output data between each epoch of training is less than a third pre-defined threshold;

training a second machine learning model by providing, as input, extracted features from the batch of macroscopic training images until an improvement in fourth output data between each epoch of training is less than a fourth pre-defined threshold;

locking the first and second convolutional neural network models and the first and second machine learning models;

generating a first set of output features from each epoch of training in the first convolutional neural network model and the first machine learning model;

generating a second set of output features from each epoch of training in the second convolutional neural network model and the second machine learning model;

obtaining a batch of electronic health record (EHR) data; and

training a fully connected network by using the first set of output features, the second set of output features, and the batch of EHR data until an improvement in fifth output data between each epoch of training is less than a fifth pre-defined threshold.