Sequential feedback ensemble model for multilabel classification
A sequential feedback prediction system predicts a risk of an event associated with a set of potential outcomes. The system receives training data including data records that have information associated with risk factors. Each data record may be labeled with outcomes that have inherent sequential dependencies. The system trains sub-models, each sub-model predicting risks for a respective outcome using a set of risk factors. For each sub-model, the system may determine a subset of sub-models whose results are used as input for the sub-model. The system determines an order to run the set of sub-models such that prediction results for preceding sub-models may be used as inputs for subsequent sub-models. The system may determine a number of rounds to run the ordered sequence of sub-models until the performance of the ensemble model meets a predetermined threshold. The system may generate a risk score for each of the set of outcomes.
1 . A method for training an ensemble model comprising:
receiving a set of training data, the set of training data including a plurality of data records for a plurality of individuals, each data record including information associated with a set of risk factors, including risk factors for high risk pregnancy comprising hypertension, diabetes or substance abuse, and labeled outcomes for a set of outcomes, the set of outcomes including preterm delivery, neonatal abstinence syndrome, neonatal intensive care unit, and C-section, the set of outcomes having inherent sequential dependencies correlated with each other;
constructing an ensemble model comprising steps:
training a set of sub-models for the set of outcomes, each sub-model trained to predict a risk associated with an outcome of the set of outcomes, wherein each sub-model predicts risks for a respective outcome using a set of risk factors corresponding to the respective outcome;
for each sub-model, determining a subset of the set of sub-models, wherein results generated by the subset of the set of sub-models are used as input for the sub-model; and
determining an order for connecting the set of sub-models, wherein each subsequent sub-model uses a set of corresponding risk factors for the sub-model and a set of predicted results from a previous sub-model as input;
constructing the ensemble model comprised of the trained set of sub-models and the order for connecting the trained set of sub-models, wherein the ensemble model is trained to generate a final prediction result or results by leveraging the results from the collection of prediction results of previous sub-models;
running the constructed ensemble model for a particular patient to predict the likelihood the particular patient will have a pregnancy with preterm delivery, neonatal abstinence syndrome, neonatal intensive care unit, or C-section; and
outputting the predicted likelihood the particular patient will have a pregnancy with preterm delivery, neonatal abstinence syndrome, neonatal intensive care unit, or C-section.
2 . The method of claim 1 , wherein the order is determined based on precision or recall of the predicted results of the set of sub-models.
3 . The method of claim 1 , further comprising:
determining risks associated with the set of outcomes by training a Natural Language Process (NLP) model based on a set of textual training data;
consolidating the risks associated with the set of outcomes from the NLP model with the predicted likelihood the particular patient will have a pregnancy with preterm delivery, neonatal abstinence syndrome, neonatal intensive care unit, or C-section;
generating a risk profile based on the predicted likelihood the particular patient will have a pregnancy with preterm delivery, neonatal abstinence syndrome, neonatal intensive care unit, or C-section predicted by the ensemble model and the risks determined by the NLP model that leverages both qualitative and quantitative analysis.
4 . The method of claim 1 , wherein each outcome of the set of outcomes is associated with a predetermined threshold, and responsive to a predicted risk score associated with an outcome exceeding the predetermine thresholds, predict that a respective individual will be associated with the outcome.
5 . The method of claim 1 , wherein the ensemble model constructs a second round of predictions for one or more sub-models of the set of sub-models.
6 . The method of claim 5 , wherein the ensemble model determines to stop including additional rounds of sub-models responsive to a metric of the ensemble model achieving a predetermined performance goal.
7 . A non-transitory computer-readable storage medium storing executable computer instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the instructions comprising instructions to:
receive a set of training data, the set of training data including a plurality of data records for a plurality of individuals, each data record including information associated with a set of risk factors, including risk factors for high risk pregnancy comprising hypertension, diabetes or substance abuse, and labeled outcomes for a set of outcomes, the set of outcomes including preterm delivery, neonatal abstinence syndrome, neonatal intensive care unit, and C-section, the set of outcomes having inherent sequential dependencies correlated with each other;
construct an ensemble model by causing the one or more processors to:
train a set of sub-models for the set of outcomes, each sub-model trained to predict a risk associated with an outcome of the set of outcomes, wherein each sub-model predicts risks for a respective outcome using a set of risk factors corresponding to the respective outcome;
for each sub-model, determine a subset of the set of sub-models, wherein results generated by the subset of the set of sub-models are used as input for the sub-model;
determining an order for connecting the set of sub-models, wherein each subsequent sub-model uses a set of corresponding risk factors for the sub-model and a set of predicted results from a previous sub-model as input; and
constructing the ensemble model comprised of the trained set of sub-models and the order for connecting the trained set of sub-models, wherein the ensemble model is trained to generate a final prediction result or results by leveraging the results from the collection of prediction results of previous sub-models;
running the constructed ensemble model for a particular patient to predict the likelihood the particular patient will have a pregnancy with preterm delivery, neonatal abstinence syndrome, neonatal intensive care unit, or C-section; and
outputting the predicted likelihood the particular patient will have a pregnancy with preterm delivery, neonatal abstinence syndrome, neonatal intensive care unit, or C-section.
8 . The non-transitory computer-readable storage medium of claim 7 , wherein the order is determined based on precision or recall of the predicted results of the set of sub-models.
9 . The non-transitory computer-readable storage medium of claim 7 , wherein constructing an ensemble model further comprises:
determining risks associated with the set of outcomes by training a Natural Language Process (NLP) model based on a set of textual training data;
consolidating the risks associated with the set of outcomes from the NLP model with the predicted likelihood the particular patient will have a pregnancy with preterm delivery, neonatal abstinence syndrome, neonatal intensive care unit, or C-section;
generating a risk profile based on the predicted likelihood the particular patient will have a pregnancy with preterm delivery, neonatal abstinence syndrome, neonatal intensive care unit, or C-section predicted by the ensemble model and the risks determined by the NLP model that leverages both qualitative and quantitative analysis.
10 . The non-transitory computer-readable storage medium of claim 7 , wherein each outcome of the set of outcomes is associated with a predetermined threshold, and responsive to a predicted risk score associated with an outcome exceeding the predetermine thresholds, predict that a respective individual will be associated with the outcome.
11 . The non-transitory computer-readable storage medium of claim 7 , wherein the ensemble model constructs a second round of predictions for one or more sub-models of the set of sub-models.
12 . The non-transitory computer-readable storage medium of claim 11 , wherein the ensemble model determines to stop including additional rounds of sub-models responsive to a metric of the ensemble model achieving a predetermined performance goal.
13 . The method of claim 1 , further comprising the steps of:
after constructing the ensemble model, generating, for an individual, a comprehensive risk profile by: (i) applying an natural language process (NLP) model trained on textual data associated with the individual to determine risks associated with the individual; (ii) consolidating the risks determined by the NLP model with risk scores generated by the ensemble model for the set of outcomes; and (iii) producing member-level quantitative explanations and visualizations that identify risk factors driving the predicted risks.
14 . The non-transitory computer-readable storage medium of claim 7 , having additional executable computer instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the instructions comprising instructions to:
after constructing the ensemble model, generating, for an individual, a comprehensive risk profile by: (i) applying an natural language process (NLP) model trained on textual data associated with the individual to determine risks associated with the individual; (ii) consolidating the risks determined by the NLP model with risk scores generated by the ensemble model for the set of outcomes; and (iii) producing member-level quantitative explanations and visualizations that identify risk factors driving the predicted risks.