Systems and methods for transforming and storing data from multiple studies
There is disclosed a method and system for combining datasets. Study results may be retrieved. Each study result may include datapoints. Each datapoint may include attributes. Study questions may be extracted from the study results. The questions may be converted into a standardized format. Categories may be assigned to the questions. The questions may be grouped together into groups. A response scale may be determined for each of the groups. The responses may be rescaled using the corresponding response scale. A final dataset may be generated by combining the study results. Features may be selected using the final dataset. A machine learning algorithm may be trained using the features of the final dataset.
1 . A method comprising:
retrieving a plurality of study results, wherein each study result comprises a plurality of datapoints, wherein each datapoint corresponds to an individual, and wherein each datapoint comprises a plurality of attributes of the individual and outcome data corresponding to a treatment received by the individual;
extracting a plurality of questions from the plurality of study results, wherein the plurality of questions comprises questions from questionnaires given to individuals enrolled in each study of the plurality of studies;
converting each question of the plurality of questions into questions in a standardized format;
assigning one or more categories to each of the plurality of questions;
determining, based on the one or more categories assigned to each question, a plurality of groups of questions;
determining a response scale to use for each group of the plurality of groups;
rescaling responses to the groups of questions based on the respective response scale of the group;
combining questions for each group of the plurality of groups;
imputing missing attribute values in the plurality of study results;
generating a final dataset by combining the plurality of study results;
assigning a label to each datapoint of the final dataset, wherein the label indicates the outcome data of the individual corresponding to the respective datapoint;
splitting the final dataset into a training set, validation set, and testing set;
selecting, based on the training set, a plurality of features in the training set; and
training a machine learning algorithm (MLA) comprising a neural network using the plurality of features of the training set, wherein the training comprises:
encoding, by an encoder, values of the plurality of features to form vectors for each datapoint of the training set,
inputting the vectors to the MLA,
outputting, by the MLA, a predicted efficacy of a treatment for each datapoint of the training set,
determining an amount of loss for each datapoint of the training set using a loss function that compares the predicted efficacy of the treatment for each datapoint of the training set to the label for the respective datapoint, and
adjusting the neural network to reduce the amount of loss between the predicted efficacy of the treatment for each datapoint of the training set and the label for the respective datapoint, thereby optimizing the predicted efficacy of the treatment output by the MLA;
receiving questionnaire responses corresponding to an individual;
inputting the questionnaire responses into the MLA;
outputting, by the MLA, a predicted efficacy of the treatment for the individual;
generating, based on the predicted efficacy of the treatment, an interface; and
outputting for display the interface.
2 . The method of claim 1 , wherein the questionnaires comprise clinical questionnaires given to patients enrolled in clinical trials.
3 . The method of claim 1 , wherein each datapoint corresponds to a patient enrolled in a clinical trial, and wherein the plurality of attributes comprise an indication of a treatment given to the patient.
4 . The method of claim 3 , wherein the plurality of attributes comprise an indication of responses to a clinical questionnaire given to the patient.
5 . The method of claim 3 , wherein the plurality of attributes comprise an indication of whether the treatment is effective for the patient.
6 . The method of claim 3 , wherein the plurality of attributes comprise an indication of side effects experienced by the patient.
7 . The method of claim 3 , wherein the plurality of attributes comprise physiological data of the patient.
8 . The method of claim 3 , wherein the plurality of attributes comprise historical data of the patient.
9 . The method of claim 3 , wherein the plurality of attributes comprise sociodemographic data of the patient.
10 . The method of claim 3 , wherein the plurality of attributes comprise psychological data of the patient.
11 . The method of claim 3 , wherein rescaling responses to the groups of questions comprises using equipercentile scaling to rescale the responses.
12 . A method comprising:
retrieving a plurality of study results, wherein each study result comprises a plurality of datapoints, wherein each datapoint corresponds to an individual, and wherein each datapoint comprises a plurality of attributes of the individual and outcome data corresponding to a treatment received by the individual;
extracting a plurality of questions from the plurality of study results, wherein the plurality of questions comprises questions from questionnaires given to individuals enrolled in each study of the plurality of studies;
converting each question of the plurality of questions into questions in a standardized format;
assigning one or more categories to each of the plurality of questions;
determining a group of questions, of the plurality of questions, that have a same set of assigned categories;
combining the group of questions;
generating a final dataset by combining the plurality of study results;
assigning a label to each datapoint of the final dataset, wherein the label indicates the outcome data of the individual corresponding to the respective datapoint;
selecting a plurality of features in the final dataset; and
training a machine learning algorithm (MLA) comprising a neural network using the plurality of features, wherein the training comprises:
encoding, by an encoder, values of the plurality of features to form vectors for each datapoint of the training set,
inputting the vectors to the MLA,
outputting, by the MLA, a predicted efficacy of a treatment for each datapoint of the training set,
determining an amount of loss for each datapoint of the training set using a loss function that compares the predicted efficacy of the treatment for each datapoint of the training set to the label for the respective datapoint, and
adjusting the neural network to reduce the amount of loss between the predicted efficacy of the treatment for each datapoint of the training set and the label for the respective datapoint, thereby optimizing the predicted efficacy of the treatment output by the MLA;
receiving questionnaire responses corresponding to an individual;
inputting the questionnaire responses into the MLA;
outputting, by the MLA, a predicted efficacy of the treatment for the individual;
generating, based on the predicted efficacy of the treatment, an interface; and
outputting for display the interface.
13 . The method of claim 12 , wherein combining the group of questions comprises:
determining a response scale for the group of questions; and
rescaling responses corresponding to the group of questions based on the response scale.
14 . The method of claim 12 , wherein the questionnaires comprise clinical questionnaires given to patients enrolled in clinical trials.
15 . The method of claim 12 , wherein each datapoint corresponds to a patient enrolled in a clinical trial, and wherein the plurality of attributes comprise an indication of a treatment given to the patient.
16 . The method of claim 15 , wherein the plurality of attributes comprise an indication of whether the treatment is effective for the patient.
17 . The method of claim 15 , wherein the plurality of attributes comprise an indication of side effects experienced by the patient.
18 . A method for predicting treatment efficacy for a patient, the method comprising:
training a machine learning algorithm (MLA) comprising a neural network by:
retrieving a plurality of study results corresponding to a plurality of treatments, wherein each study result comprises a plurality of datapoints, wherein each datapoint corresponds to an individual, and wherein each datapoint comprises a plurality of attributes of the individual and outcome data corresponding to a treatment received by the individual,
extracting a plurality of questions from the plurality of study results, wherein the plurality of questions comprise questions from questionnaires given to individuals enrolled in each study of the plurality of studies,
converting each question of the plurality of questions into questions in a standardized format,
assigning one or more categories to each of the plurality of questions,
determining a group of questions, of the plurality of questions, that have a same set of assigned categories,
combining the group of questions,
generating a final dataset by combining the plurality of study results,
assigning a label to each datapoint of the final dataset, wherein the label indicates the outcome data of the individual corresponding to the respective datapoint; and
training the MLA using the final dataset, wherein the training comprises:
encoding, by an encoder, values of each datapoint of the training set to form vectors,
inputting the vectors to the MLA,
outputting, by the MLA, a predicted efficacy of a treatment for each datapoint of the training set,
determining an amount of loss for each datapoint of the training set using a loss function that compares the predicted efficacy of the treatment for each datapoint of the training set to the label for the respective datapoint, and
adjusting the neural network to reduce the amount of loss between the predicted efficacy of the treatment for each datapoint of the training set and the label for the respective datapoint, thereby optimizing the predicted efficacy of the treatment output by the MLA;
receiving questionnaire responses from the patient;
inputting the questionnaire responses into the MLA;
outputting, by the MLA, a predicted efficacy of each of the plurality of treatments;
generating, based on the predicted efficacy of each of the plurality of treatments, an interface; and
outputting for display the interface.
19 . The method of claim 18 , wherein the plurality of attributes comprise an indication of side effects experienced by the patient.
20 . The method of claim 18 , wherein the interface displays one or more treatment options and a prediction corresponding to each of the one or more treatment options.