Data subset selection for federated learning
Second machine learning models trained using respective second data sets can be received. The second machine learning models can be run using a first data set used in training a first machine learning model, where the second machine learning models produce respective outputs. Scores associated with the second machine learning models can be determined by comparing the respective outputs with ground truth associated with the first data set. Based on the scores associated with the second machine learning models, whether the first data set is to be discarded or kept can be determined for training the first machine learning model.
1 . A system comprising:
a processor; and
a memory device coupled with the processor;
the processor configured to:
receive second machine learning models trained using respective second data sets;
run the second machine learning models using a first data set which was used in training a first machine learning model, the second machine learning models producing respective outputs;
determine scores associated with the second machine learning models by comparing the respective outputs with ground truth associated with the first data set, wherein each of the scores indicates how often a respective second machine learning model provides a right prediction using the first data set, in predicting an outcome the second machine learning models and the first machine learning model have been trained to predict;
based on the scores associated with the second machine learning models, determine which training data in the first data set is to be discarded or kept for training the first machine learning model,
wherein the first machine learning model checks the first data set using each of the second machine learning models, and each of the first data set and the respective second data sets has different data quality; and
sharing the scores with processors that run the second machine learning models without sharing the first data set used in training the first machine learning model with the second machine learning models.
2 . The system of claim 1 , wherein the first machine learning model and the second machine learning models are distributed across different computer systems and the first machine learning model is trained independently from the second machine learning models, wherein the first data set and the respective second data sets are unshared among the different computer systems.
3 . The system of claim 1 , wherein an ensemble model is built based on at least the scores associated with the second machine learning models, wherein which training data in the first data set is to be discarded or kept for training the first machine learning model is determined by running the ensemble model using the first data set and comparing an output of the ensemble model with the ground truth associated with the first data set.
4 . The system of claim 1 , wherein the processor is further configured to receive an ensemble model built based on at least the scores associated with the second machine learning models, run the ensemble model using the first data set, and select the training data from the first data set having ground truth that matches with an output of the ensemble model.
5 . The system of claim 1 , wherein the second machine learning models are weighted based on the scores associated with the second machine learning models.
6 . The system of claim 1 , wherein the processor is further configured to train the first machine learning model using the training data in the first data set determined to be kept.
7 . The system of claim 1 , wherein the processor is further configured to exchange the first machine learning model trained using the first data set with a plurality of sites running the second machine learning models.
8 . A computer-implemented method comprising:
receiving second machine learning models trained using respective second data sets;
running the second machine learning models using a first data set which was used in training a first machine learning model, the second machine learning models producing respective outputs;
determining scores associated with the second machine learning models by comparing the respective outputs with ground truth associated with the first data set, wherein each of the scores indicates how often a respective second machine learning model provides a right prediction using the first data set, in predicting an outcome the second machine learning models and the first machine learning model have been trained to predict;
based on the scores associated with the second machine learning models, determining which training data in the first data set is to be discarded or kept for training the first machine learning model,
wherein the first machine learning model checks the first data set using each of the second machine learning models, and each of the first data set and the respective second data sets has different data quality; and
sharing the scores with processors that run the second machine learning models without sharing the first data set used in training the first machine learning model with the second machine learning models.
9 . The computer-implemented method of claim 8 , wherein the first machine learning model and the second machine learning models are distributed across different computer systems and the first machine learning model is trained independently from the second machine learning models, wherein the first data set and the respective second data sets are unshared among the different computer systems.
10 . The computer-implemented method of claim 8 , wherein an ensemble model is built based on at least the scores associated with the second machine learning models, wherein which training data in the first data set is to be discarded or kept for training the first machine learning model is determined by running the ensemble model using the first data set and comparing an output of the ensemble model with the ground truth associated with the first data set.
11 . The computer-implemented method of claim 8 , wherein the method further includes receiving an ensemble model built based on at least the scores associated with the second machine learning models, running the ensemble model using the first data set, and selecting the training data in the first data set having ground truth that matches with an output of the ensemble model.
12 . The computer-implemented method of claim 8 , wherein the second machine learning models are weighted based on the scores associated with the second machine learning models.
13 . The computer-implemented method of claim 8 , wherein the method further includes training the first machine learning model using the training data in first data set determined to be kept.
14 . The computer-implemented method of claim 8 , wherein the method further includes exchanging the first machine learning model trained using the first data set with a plurality of sites respectively running the second machine learning models.
15 . A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions readable by a device to cause the device to:
receive second machine learning models trained using respective second data sets;
run the second machine learning models using a first data set which was used in training a first machine learning model, the second machine learning models producing respective outputs;
determine scores associated with the second machine learning models by comparing the respective outputs with ground truth associated with the first data set, wherein each of the scores indicates how often a respective second machine learning model provides a right prediction using the first data set, in predicting an outcome the second machine learning models and the first machine learning model have been trained to predict;
based on the scores associated with the second machine learning models, determine which training data in the first data set is to be discarded or kept for training the first machine learning model,
wherein the first machine learning model checks the first data set using each of the second machine learning models, and each of the first data set and the respective second data sets has different data quality; and
sharing the scores with processors that run the second machine learning models without sharing the first data set used in training the first machine learning model with the second machine learning models.
16 . The computer program product of claim 15 , wherein the first machine learning model and the second machine learning models are distributed across different computer systems and the first machine learning model is trained independently from the second machine learning models, wherein the first data set and the respective second data sets are unshared among the different computer systems.
17 . The computer program product of claim 15 , wherein an ensemble model is built based on at least the scores associated with the second machine learning models, wherein which training data in the first data set is to be discarded or kept for training the first machine learning model is determined by running the ensemble model using the first data set and comparing an output of the ensemble model with the ground truth associated with the first data set.
18 . The computer program product of claim 15 , wherein the device is further caused to receive an ensemble model built based on at least the scores associated with the second machine learning models, run the ensemble model using the first data set, and select the training data from the first data set having ground truth that matches with an output of the ensemble model.
19 . The computer program product of claim 15 , wherein the second machine learning models are weighted based on the scores associated with the second machine learning models.
20 . The computer program product of claim 15 , wherein the device is further caused to train the first machine learning model using the training data in the first data set determined to be kept, and to exchange the first machine learning model trained with a plurality of sites respectively running the second machine learning models.