Model routing and robust outlier detection

1 . A computer-implemented method for using a clinical study tool to assess a clinical study based on a plurality of clinical-study metrics, the method comprising:

receiving, via graphical user interface of the clinical study tool, a user input comprising one or more user selected sub-categories from one or more categories associated with the clinical study;

obtaining a model performance lookup table comprising performance benchmarks for each of a plurality of category-specific machine-learning models with respect to a baseline machine-learning model, wherein the model performance lookup table is generated by:

obtaining test inputs and corresponding ground-truth values for one or more clinical-study metrics;

evaluating, based on the test inputs and the corresponding ground-truth values, the baseline machine-learning model and the plurality of category-specific machine-learning models; and

generating the performance benchmarks based on the evaluation;

selecting, for each clinical-study metric of the plurality of clinical-study metrics, a model from a set of machine-learning models comprising the baseline machine-learning model and the plurality of category-specific machine-learning models wherein the selection is based on the model performance lookup table;

predicting values for the plurality of clinical-study metrics using corresponding selected models;

generating one or more predictions for assessing the clinical study based on the predicted values for the plurality of clinical-study metrics; and

displaying, via the graphical user interface of the clinical study tool, the one or more predictions for assessing the clinical study.

2 . The method of claim 1 , wherein selecting the model from either the baseline machine-learning model or one of the plurality of category-specific machine-learning models to predict the clinical-study metric using the model performance lookup table comprises:

comparing past performances of the baseline machine-learning model and one or more category-specific machine-learning models associated with the user selected sub-categories in predicting the clinical-study metric;

determining, based on the comparison, that the past performance of the baseline machine-learning model is better than the one or more category-specific machine-learning models; and

in response to the determination, selecting the baseline machine-learning model for predicting the clinical-study metric.

3 . The method of claim 1 , wherein selecting the model from either the baseline machine-learning model or one of the plurality of category-specific machine-learning models to predict the clinical-study metric using the model performance lookup table comprises:

comparing past performances of the baseline machine-learning model and one or more category-specific machine-learning models associated with the user selected sub-categories in predicting the clinical-study metric;

determining, based on the comparison, that the past performances of the one or more category-specific machine-learning models are better than the baseline machine-learning model; and

in response to the determination, selecting one of the category-specific machine-learning models that has a past performance relatively higher than other category-specific machine-learning models in predicting the clinical-study metric.

4 . The method of claim 1 , further comprising training the baseline machine-learning model and the plurality of category-specific machine-learning models, wherein the training comprises:

receiving a training dataset comprising ground-truth values for the plurality of clinical-study metrics for the one or more categories;

filtering the training dataset by each sub-category of the one or more categories to obtain a plurality of category-specific training datasets;

training the baseline machine-learning model using the training dataset prior to filtering; and

training the plurality of category-specific machine-learning models using the plurality of category-specific training datasets.

5 . The method of claim 4 , wherein the model performance lookup table is generated after training of the baseline machine-learning model and the plurality of category-specific machine-learning models.

6 . The method of claim 4 , further comprising deploying the trained baseline machine-learning model and the plurality of category-specific machine-learning models for using at inference time for clinical studies, wherein the deploying comprises:

integrating the trained baseline machine-learning model and the plurality of category-specific machine-learning models into a container or framework; and

generating a prediction application programming interface (API) endpoint for the container or framework.

7 . The method of claim 6 , further comprising:

in response to receiving the user input, making a call to the prediction API endpoint.

8 . The method of claim 6 , further comprising:

after deploying the trained baseline machine-learning model and the plurality of category-specific machine-learning models into production, validating the plurality of category-specific machine-learning models for testing an accuracy in the predictions.

9 . The method of claim 8 , wherein validating the plurality of category-specific machine-learning models comprises:

for each clinical-study metric and for each category:

predicting the clinical-study metric using the baseline machine-learning model and a first category-specific machine-learning model associated with the category;

determining an intersection between a test set of the training dataset and a test set of a dataset filtered by the category;

removing a set of samples from an intersected test set for which a model selected for predicting the clinical-study metric is the first category-specific machine-learning model;

training a second baseline machine-learning model and a plurality of second category-specific machine-learning models using remaining samples in the intersected test set;

predicting the clinical-study metric for the removed set of samples using the second baseline machine-learning model and the plurality of second category-specific machine-learning models;

comparing the prediction of the clinical-study metric, for each sample, by the first category-specific machine-learning model associated with the category, a second category-specific machine-learning model associated with the category, and the second baseline machine-learning model;

determining concordance between the first category-specific machine-learning model and the second category-specific machine-learning model based on the comparison; and

validating the first category-specific machine-learning model based on the determined concordance.

10 . The method of claim 4 , further comprising:

detecting and removing outliers from the training dataset prior to the training of the baseline machine-learning model and the plurality of category-specific machine-learning models.

11 . The method of claim 10 , wherein detecting and removing the outliers from the training dataset:

determining a particular threshold over which to remove the outliers from the training dataset;

determining a median of the training dataset;

determining a median absolute deviation of the training dataset;

calculating, for each data point within the training dataset, a score based on the median and the median absolute deviation;

detecting data points with scores with absolute values greater than the particular threshold as the outliers; and

removing the detected data points from the training dataset.

12 . The method of claim 11 , wherein determining the particular threshold comprises:

setting (1) a tolerance limit for data loss when removing the outliers from the training dataset and (2) a threshold-adjustment value by which to adjust a current threshold for removing the outliers;

removing a first set of outliers from the training dataset using a first scoring technique;

determining a first amount of the training dataset remaining after the removal of the first set of outliers;

determining, through an iterative process, a final adjusted threshold for removing the outliers from the training dataset, wherein an initial iteration in the iterative process comprises:

removing a second set of outliers from the training dataset using a second scoring technique and the current threshold;

determining a second amount of the training dataset remaining after the removal of the second set of outliers;

determining an amount of data loss in the training dataset by comparing the first amount of the training dataset and the second amount of the training dataset; and

adjusting the current threshold by the threshold-adjustment value if the determined amount of data loss is greater than the set tolerance limit for data loss; and

using the final adjusted threshold as the particular threshold for removing the outliers from the training dataset.

13 . The method of claim 1 , wherein evaluating, based on the test inputs and the corresponding ground-truth values, the baseline machine-learning model and the category-specific machine-learning models comprises:

obtaining, using the test inputs, predicted values for the plurality of clinical-study metrics by the baseline machine-learning model and the plurality of category-specific machine-learning models;

comparing the predicted values for the plurality of clinical-study metrics by each model with the corresponding ground-truth values; and

measuring, for each of the baseline machine-learning model and the plurality of category-specific machine-learning models, differences between the predicted values and the corresponding ground-truth values; and

wherein generating the performance benchmarks based on the evaluation comprises:

generating a measure of model performance for each category-specific machine-learning model with respect to the baseline machine-learning model based on the measured differences; and

storing the measure of model performance for each category-specific machine-learning model with respect to the baseline machine-learning model in a cell corresponding to each clinical-study metric for each category in the model performance lookup table.

14 . The method of claim 1 , wherein the one or more predictions for assessing the clinical study comprises:

operational burden measures;

patient burden measures; or

site burden measures.

15 . The method of claim 1 , wherein a clinical-study metric of the plurality of clinical-study metrics is one of:

an enrollment rate of patients;

a screening rate of patients;

a screen failure ratio;

a patient dropout ratio;

time to regulatory approval;

time to enrollment completion;

time to first patient enrollment;

protocol deviations per patient; or

number of unique procedures in the clinical study.

16 . The method of claim 1 , wherein the one or more categories comprise:

a first category comprising different phases of a clinical trial or the clinical study; and

a second category comprising different therapeutic areas in the clinical study.

17 . The method of claim 16 , wherein each subcategory of the first category is one of the different phases of the clinical trial.

18 . The method of claim 16 , wherein each subcategory of the second category is one of the therapeutic areas in the clinical study.

19 . A computer-program product for using a clinical study tool to assess a clinical study based on a plurality of clinical-study metrics, the computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to:

receive, via graphical user interface of the clinical study tool, a user input comprising one or more user selected sub-categories from one or more categories associated with the clinical study;

obtain a model performance lookup table comprising performance benchmarks for each of a plurality of category-specific machine-learning models with respect to a baseline machine-learning model, wherein the model performance lookup table is generated by:

obtaining test inputs and corresponding ground-truth values for one or more clinical-study metrics;

evaluating, based on the test inputs and the corresponding ground-truth values, the baseline machine-learning model and category-specific machine-learning models; and

generating the performance benchmarks based on the evaluation;

select, for each clinical-study metric of the plurality of clinical-study metrics, a model from a set of machine-learning models comprising the baseline machine-learning model and the plurality of category-specific machine-learning models wherein the selection is based on the model performance lookup table;

predict values for the plurality of clinical-study metrics using corresponding selected models;

generate one or more predictions for assessing the clinical study based on the predicted values for the plurality of clinical-study metrics; and

display, via the graphical user interface of the clinical study tool, the one or more predictions for assessing the clinical study.

20 . A system for using a clinical study tool to assess a clinical study based on a plurality of clinical-study metrics, comprising:

one or more processors; and

a non-transitory computer readable storage medium containing instructions which, when executed on the one or more processors, cause the one or more processors to:

receive, via graphical user interface of the clinical study tool, a user input comprising one or more user selected sub-categories from one or more categories associated with the clinical study;

obtain a model performance lookup table comprising performance benchmarks for each of a plurality of category-specific machine-learning models with respect to a baseline machine-learning model, wherein the model performance lookup table is generated by:

obtaining test inputs and corresponding ground-truth values for one or more clinical-study metrics;

evaluating, based on the test inputs and the corresponding ground-truth values, the baseline machine-learning model and category-specific machine-learning models; and

generating the performance benchmarks based on the evaluation;

select, for each clinical-study metric of the plurality of clinical-study metrics, a model from a set of machine-learning models comprising the baseline machine-learning model and the plurality of category-specific machine-learning models wherein the selection is based on the model performance lookup table;

predict values for the plurality of clinical-study metrics using corresponding selected models;

generate one or more predictions for assessing the clinical study based on the predicted values for the plurality of clinical-study metrics; and

display, via the graphical user interface of the clinical study tool, the one or more predictions for assessing the clinical study.