DISTRIBUTED AND FEDERATED LEARNING USING MULTI-LAYER MACHINE LEARNING MODELS
In one set of embodiments, a computing node in a plurality of computing nodes can train a first ML model on a local training dataset comprising a plurality of labeled training data instances, where the training is performed using a distributed/federated training approach across the plurality of computing nodes and where the training results in a trained version of the first ML model. The computing node can further compute, using the trained version of the first ML model, a training value measure for each labeled training data instance in the local training dataset and identify a subset of the plurality of labeled training data instances based on the computed training value measures. The computing node can then train a second ML model on the subset, where the training of the second ML model is performed using the distributed/federated training approach.
1 . A method comprising:
training, by a computing node in a plurality of computing nodes, a first machine learning (ML) model on a local training dataset comprising a plurality of labeled training data instances, wherein the training of the first ML model is performed using a distributed or federated training approach across the plurality of computing nodes, and wherein the training of the first ML model results in a trained version of the first ML model;
computing, by the computing node using the trained version of the first ML model, a training value measure for each labeled training data instance in the local training dataset, the training value measure indicating a degree of usefulness of the labeled training data instance for ML training;
identifying, by the computing node, a subset of the plurality of labeled training data instances in the local training dataset based at least in part on the computed training value measures; and
training, by the computing node, a second ML model on the subset, wherein the training of the second ML model is performed using the distributed or federated training approach across the plurality of computing nodes, and wherein the training of the second ML model results in a trained version of the second ML model.
2 . The method of claim 1 wherein the second ML model is larger or more complex in structure than the first ML model.
3 . The method of claim 1 wherein computing the training value measure for each labeled training data instance comprises:
generating, using the trained version of the first ML model, a prediction for the labeled training data instance; and
computing the training value measure as a function of a distance between the prediction and a label of the labeled training data instance.
4 . The method of claim 1 wherein identifying the subset comprises:
transmitting first statistics regarding the computed training value measures to other computing nodes in the plurality of computing nodes.
5 . The method of claim 4 wherein identifying the subset further comprises:
receiving, from said other computing nodes, second statistics regarding training value measures computed by said other computing nodes; and
identifying the subset based on the first statistics and the second statistics.
6 . The method of claim 5 wherein the transmitting of the first statistics and the receiving of the second statistics is performed using secure multi-party computation (MPC).
7 . The method of claim 1 further comprising:
receiving a query data instance;
generating, via the trained version of the first ML model, a first prediction for the query data instance and a confidence level for the first prediction;
if the confidence level for the first prediction meets or exceeds a threshold, outputting the first prediction as a final prediction result for the query data instance; and
if the confidence level for the first prediction does not meet or exceed the threshold:
generating, via the trained version of the second ML model, a second prediction for the query data instance; and
outputting the second prediction as the final prediction result for the query data instance.
8 . A non-transitory computer readable storage medium having stored thereon program code executable by a computing node in a plurality of computing nodes, the program code causing the computer system to execute a method comprising:
training a first machine learning (ML) model on a local training dataset comprising a plurality of labeled training data instances, wherein the training of the first ML model is performed using a distributed or federated training approach across the plurality of computing nodes, and wherein the training of the first ML model results in a trained version of the first ML model;
computing, using the trained version of the first ML model, a training value measure for each labeled training data instance in the local training dataset, the training value measure indicating a degree of usefulness of the labeled training data instance for ML training;
identifying a subset of the plurality of labeled training data instances in the local training dataset based at least in part on the computed training value measures; and
training a second ML model on the subset, wherein the training of the second ML model is performed using the distributed or federated training approach across the plurality of computing nodes, and wherein the training of the second ML model results in a trained version of the second ML model.
9 . The non-transitory computer readable storage medium of claim 8 wherein the second ML model is larger or more complex in structure than the first ML model.
10 . The non-transitory computer readable storage medium of claim 8 wherein computing the training value measure for each labeled training data instance comprises:
generating, using the trained version of the first ML model, a prediction for the labeled training data instance; and
computing the training value measure as a function of a distance between the prediction and a label of the labeled training data instance.
11 . The non-transitory computer readable storage medium of claim 8 wherein identifying the subset comprises:
transmitting first statistics regarding the computed training value measures to other computing nodes in the plurality of computing nodes.
12 . The non-transitory computer readable storage medium of claim 11 wherein identifying the subset further comprises:
receiving, from said other computing nodes, second statistics regarding training value measures computed by said other computing nodes; and
identifying the subset based on the first statistics and the second statistics.
13 . The non-transitory computer readable storage medium of claim 12 wherein the transmitting of the first statistics and the receiving of the second statistics is performed using secure multi-party computation (MPC).
14 . The non-transitory computer readable storage medium of claim 8 wherein the method further comprises:
receiving a query data instance;
generating, via the trained version of the first ML model, a first prediction for the query data instance and a confidence level for the first prediction;
if the confidence level for the first prediction meets or exceeds a threshold, outputting the first prediction as a final prediction result for the query data instance; and
if the confidence level for the first prediction does not meet or exceed the threshold:
generating, via the trained version of the second ML model, a second prediction for the query data instance; and
outputting the second prediction as the final prediction result for the query data instance.
15 . A computing node comprising:
a processor; and
a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
train a first machine learning (ML) model on a local training dataset comprising a plurality of labeled training data instances, wherein the training of the first ML model is performed using a distributed or federated training approach across a plurality of computing nodes including the computing node, and wherein the training of the first ML model results in a trained version of the first ML model;
compute, using the trained version of the first ML model, a training value measure for each labeled training data instance in the local training dataset, the training value measure indicating a degree of usefulness of the labeled training data instance for ML training;
identify a subset of the plurality of labeled training data instances in the local training dataset based at least in part on the computed training value measures; and
train a second ML model on the subset, wherein the training of the second ML model is performed using the distributed or federated training approach across the plurality of computing nodes, and wherein the training of the second ML model results in a trained version of the second ML model.
16 . The computing node of claim 15 wherein the second ML model is larger or more complex in structure than the first ML model.
17 . The computing node of claim 15 wherein the program code that causes the processor to compute the training value measure for each labeled data instance comprises program code that causes the processor to:
generate, using the trained version of the first ML model, a prediction for the labeled training data instance; and
compute the training value measure as a function of a distance between the prediction and a label of the labeled training data instance.
18 . The computing node of claim 15 wherein the program code that causes the processor to identify the subset comprises program code that causes the processor to:
transmit first statistics regarding the computed training value measures to other computing nodes in the plurality of computing nodes.
19 . The computing node of claim 18 wherein the program code that causes the processor to identify the subset further comprises program code that causes the processor to:
receive, from said other computing nodes, second statistics regarding training value measures computed by said other computing nodes; and
identify the subset based on the first statistics and the second statistics.
20 . The computing node of claim 19 wherein the transmitting of the first statistics and the receiving of the second statistics is performed using secure multi-party computation (MPC).
21 . The computing node of claim 15 wherein the program code further causes the processor to:
receive a query data instance;
generate, via the trained version of the first ML model, a first prediction for the query data instance and a confidence level for the first prediction;
if the confidence level for the first prediction meets or exceeds a threshold, output the first prediction as a final prediction result for the query data instance; and
if the confidence level for the first prediction does not meet or exceed the threshold:
generate, via the trained version of the second ML model, a second prediction for the query data instance; and
output the second prediction as the final prediction result for the query data instance.