Multimodal learning framework for analysis of clinical trials

1. A method for forecasting a clinical outcome for a subject patient based on a clinical trial involving a group of patients, comprising:

obtaining patient image data for patients of the group;

extracting features from the obtained patient image data for patients of the group, wherein the features are extracted by using a scale invariant feature transform;

obtaining patient electronic health record data for patients of the group;

extracting features from the obtained electronic health record data for patients of the group;

for each patient of the group, concatenating the extracted features;

obtaining a value of the clinical outcome for patients of the group;

using the extracted features concatenated for each member of the group, and clinical outcome values obtained for each member of the group, training a random survival forest regression model that forecasts the clinical outcome;

obtaining patient image data for the subject patient;

extracting features from the obtained patient image data for the subject patient;

obtaining patient electronic health record data for the subject patient;

extracting features from the obtained electronic health record data for the subject patient;

concatenating the extracted features for the subject patient;

applying the random forest regression model to the concatenated extracted features for the subject patient to obtain a prediction of the clinical outcome for the subject patient; and

causing a treatment plan of the subject patient to be altered based, at least in part, on the prediction of the clinical outcome for the subject patient, the alteration being with respect to operation of a treatment device.

2. The method of claim 1 wherein the clinical outcome is overall survival time.

3. The method of claim 1 wherein the clinical outcome is global risk score.

4. The method of claim 1 wherein the subject patient is a human patient.

5. The method of claim 1 wherein the clinical outcome is a cancer survival outcome.

6. One or more storage devices collectively storing a medical outcome prediction model data structure, the data structure comprising:

a trained statistical model that can be applied to image data and electronic health record data for a patient to predict a cancer survival outcome for the patient, wherein the trained statistical model is a random survival forest regression model, wherein missing image data and missing electronic health record data is imputed by using multivariate imputation by chained equations, wherein the cancer survival outcome is obtained by:

obtaining patient image data for the patient;

extracting features from the obtained patient image data for the subject patient;

obtaining patient electronic health record data for the subject patient;

extracting features from the obtained electronic health record data for the subject patient;

concatenating the extracted features for the subject patient; and

applying the random forest survival regression model to the concatenated extracted features for the subject patient; and

user interface data adapted to present the cancer survival outcome for the patient and obtain user input as a basis for altering a treatment plan for the patient, the alteration being with respect to operation of a treatment device.

7. The storage devices of claim 6 wherein the model is a deep learning-based classifier.

8. The storage devices of claim 6 wherein the model is applied to patient image data via a tumor average intensity feature.

9. The storage devices of claim 6 wherein the model is applied to patient image data via a tumor intensity histogram feature.

10. The storage devices of claim 6 wherein the model is applied to patient image data via a tumor volume feature.

11. The storage devices of claim 6 wherein the model is applied to patient image data via a scale invariant feature transform feature.

12. The storage devices of claim 6 wherein the model is applied to patient image data via a speeded-up robust feature.

13. One or more hardware storage devices collectively having contents configured to cause a computing system to perform a method for predicting a clinical outcome for a subject patient based on a clinical trial involving a group of patients, the method comprising:

obtaining patient image data for patients of the group;

extracting features from the obtained patient image data for patients of the group, wherein the features are extracted by using speeded-up robust features;

obtaining patient electronic health record data for patients of the group;

extracting features from the obtained electronic health record data for patients of the group;

for each patient of the group, concatenating the extracted features;

obtaining a value of the clinical outcome for patients of the group;

using the extracted features concatenated for each member of the group, and clinical outcome values obtained for each member of the group, training a statistical model to predict the clinical outcome;

obtaining patient image data for the subject patient;

extracting features from the obtained patient image data for the subject patient;

obtaining patient electronic health record data for the subject patient;

extracting features from the obtained electronic health record data for the subject patient;

concatenating the extracted features for the subject patient;

identifying one or more treatment options for the subject patient;

applying the random survival forest regression model to the concatenated extracted features for the subject patient and the one or more treatment options for the subject patient to obtain a prediction of the clinical outcome for each of the treatment options for the subject patient; and

causing the prediction of the clinical outcome for each of the treatment options for the subject patient to be presented to a user of the computing system; and

in response to user input, causing a treatment plan of the subject patient to be altered based, at least in part, on the user input and the prediction of the clinical outcome for each of the treatment options for the subject patient, the alteration being with respect to operation of a treatment device.

14. The one or more hardware storage devices of claim 13 wherein the extracting features from the obtained patient image data for each patient comprises:

in each of one or more images from the patient, segmenting a tumor region and/or a pancreatic region; and

performing visual analysis of the segmented region or regions to obtain a feature value.

15. The one or more hardware storage devices of claim 13 , the method further comprising:

for a distinguished patient, among patient image data for the patient and patient electronic health record data for the patient, identifying missing data;

determining imputed data for the patient corresponding to the identified missing data; and

adding the determined imputed data to the data for the distinguished patient to obtain enhanced data for the distinguished patient, and

wherein features are extracted from the obtained enhanced data for the distinguished patient.

16. The one or more hardware storage devices of claim 13 wherein multivariate imputation by chained equations is used to determine the imputed data.

17. The one or more hardware storage devices of claim 13 wherein the trained statistical model is a random forest survival regression model.

18. The one or more hardware storage devices of claim 13 wherein the trained statistical model is a deep learning-based classifier.