Systems and methods for federated validation of models

1 . A system for validating a machine learning model using federated machine learning process, the system comprising:

one or more processors; and

a non-transitory computer-readable storage medium storing instructions, which when executed by the one or more processors cause the one or more processors to perform operations comprising:

receiving, from a plurality of client devices, a plurality of sets of machine learning model data, wherein the machine learning model data in each set comprises corresponding updated model parameters resulting from training each machine learning model at a corresponding client device of the plurality of client devices using a training dataset stored at each corresponding client device of the plurality of client devices;

in response to receiving the plurality of sets of the machine learning model data, combining each set of the updated model parameters into an aggregated set of model parameters;

transmitting to each client device a first request to update a corresponding machine learning model using the aggregated set of model parameters;

in response to receiving a confirmation from each of the plurality of client devices that the corresponding machine learning model has been updated, transmitting a second request to each of the plurality of client devices to validate the machine learning model using a validation dataset, wherein each client device of the plurality of client devices validates the corresponding machine learning model using a corresponding validation dataset generated on each client device;

in response to the second request, receiving a corresponding performance score from each of the plurality of client devices, wherein each performance score indicates an accuracy level of the corresponding machine learning model; and

in response to determining that a first machine learning model corresponding to a first client device is associated with a performance score lower than a threshold, reverting a corresponding set of updated model parameters to an original set of model parameters.

2 . The system of claim 1 , wherein the instructions cause the one or more processors to transmit, to each of the plurality of client devices, a first command to split training data into the training dataset and the validation dataset, wherein the first command comprises (1) a ratio of data to be assigned to the training dataset and the validation dataset and (2) a flag for insertion into the training data to mark a portion of the training data as validation data.

3 . The system of claim 2 , wherein the instructions cause the one or more processors to transmit, to each client device in the plurality of client devices, a second command to train the corresponding machine learning model using the training dataset, wherein the second command comprises the flag for indicating the portion of the training data to skip during training.

4 . The system of claim 1 , wherein the instructions cause the one or more processors to perform operations comprising:

prior to receiving, from the plurality of client devices, the plurality of sets of machine learning model data, transmitting a request, to each of the plurality of client devices, for a corresponding original performance score, wherein each original performance score indicates a performance level of the corresponding machine learning model at the corresponding client device;

setting, for each client device, the threshold to the corresponding performance score;

comparing, for each client device of the plurality of client devices, the corresponding original performance score with the corresponding performance score for each client device; and

determining, for each client device, whether the corresponding performance score is lower than the threshold.

5 . A method for validating a machine learning model using federated machine learning process, the method comprising:

receiving, from a plurality of client devices, a plurality of sets of machine learning model data, wherein the machine learning model data in each set comprises corresponding updated model parameters resulting from training each machine learning model at a corresponding client device of the plurality of client devices;

aggregating each set of the updated model parameters into an aggregated set of model parameters;

causing each client device to update a corresponding machine learning model using the aggregated set of model parameters;

receiving, from the plurality of client devices, confirmations that the corresponding machine learning model has been updated;

causing, after receiving the confirmations that the corresponding machine learning model has been updated each of the plurality of client devices to validate the machine learning model using a validation dataset on each client device;

receiving a corresponding performance score from each of the plurality of client devices, wherein each performance score indicates an accuracy level of the corresponding machine learning model; and

in response to determining that a first machine learning model corresponding to a first client device is associated with a performance score lower than a threshold, generating a notification indicating the first client device.

6 . The method of claim 5 , further comprising in response to determining that the first machine learning model corresponding to the first client device is associated with the performance score lower than the threshold, reverting the machine learning model to an original set of model parameters.

7 . The method of claim 5 , further comprising transmitting, to each of the plurality of client devices, a first command to split training data into a training dataset and the validation dataset, wherein the first command comprises (1) a ratio of data to be assigned to the training dataset and the validation dataset (2) a flag for insertion into the training data to mark a portion of the training data as validation data.

8 . The method of claim 7 , further comprising transmitting, to each client device in the plurality of client devices, a second command to train the corresponding machine learning model using the training dataset, wherein the second command comprises the flag for indicating the portion of the training data to skip during training.

9 . The method of claim 5 , further comprising:

prior to receiving, from the plurality of client devices, the plurality of sets of machine learning model data, transmitting a request, to each of the plurality of client devices, for a corresponding original performance score, wherein each original performance score indicates a performance level of the corresponding machine learning model at the corresponding client device;

setting, for each client device, the threshold to the corresponding performance score;

comparing, for each client device of the plurality of client devices, the corresponding original performance score with the corresponding performance score for each client device; and

determining, for each client device, whether the corresponding performance score is lower than the threshold.

10 . The method of claim 9 , further comprising:

determining, for the first client device of the plurality of client devices, that a first original performance score is below a score threshold; and

in response to determining that the first original performance score is below the score threshold, adjusting a ratio of data to be assigned to a training dataset to be higher to enlarge the training dataset and to reduce the validation dataset.

11 . The method of claim 5 , further comprising transmitting a request to the first client device to train the machine learning model, wherein the request comprises a ratio of training data to be used in a training dataset and the validation dataset.

12 . The method of claim 5 , further comprising:

storing, for each client device of the plurality of client devices, a corresponding set of original performance scores and corresponding set of aggregated model parameters; and

in response to determining that the first machine learning model corresponding to the first client device is associated with the performance score lower than the threshold, selecting a first set of aggregated model parameters for the first client device, wherein the first set of aggregated model parameters is associated with a highest original performance score.

13 . One or more non-transitory, computer-readable storage media storing instructions that when executed by one or more processors cause the one or more processors to perform operations comprising:

receiving, from a client device, updated model parameters resulting from training a machine learning model at the client device;

aggregating the updated model parameters with other updated model parameters from other client devices into an aggregated set of model parameters;

causing the client device to update the machine learning model using the aggregated set of model parameters;

causing, after the client device confirms that the machine learning model has been updated, the client device to validate the machine learning model using a validation dataset, wherein the validation dataset is stored on the client device, and wherein the validation dataset is a portion of a training dataset not used in training the machine learning model;

receiving a performance score from the client device, wherein the performance score indicates an accuracy level of the machine learning model; and

in response to determining that the performance score is lower than a threshold, generating a notification indicating the client device.

14 . The one or more non-transitory, computer-readable storage media of claim 13 , wherein the instructions further cause the one or more processors to, in response to determining that the machine learning model corresponding to the client device is associated with the performance score lower than the threshold, reverting the machine learning model to an original set of model parameters.

15 . The one or more non-transitory, computer-readable storage media of claim 13 , wherein the instructions further cause the one or more processors to transmit, to each of a plurality of client devices, a first command to split training data into the training dataset and the validation dataset, wherein the first command comprises (1) a ratio of data to be assigned to the training dataset and the validation dataset (2) a flag for insertion into the training data to mark the portion of the training data as validation data.

16 . The one or more non-transitory, computer-readable storage media of claim 15 , wherein the instructions further cause the one or more processors to transmit, to each client device of the plurality of client devices, a second command to train a corresponding machine learning model using the training dataset, wherein the second command comprises the flag for indicating the portion of the training data to skip during training.

17 . The one or more non-transitory, computer-readable storage media of claim 13 , wherein the instructions further cause the one or more processors to perform operations comprising:

prior to receiving, from a plurality of client devices, a plurality of sets of machine learning model data, transmitting a request, to each of the plurality of client devices, for a corresponding original performance score, wherein each original performance score indicates a performance level of a corresponding machine learning model at a corresponding client device;

setting, for each client device, the threshold to the corresponding original performance score;

comparing, for each client device of the plurality of client devices, the corresponding original performance score with the corresponding original performance score for each client device; and

determining, for each client device, whether the corresponding original performance score is lower than the threshold.

18 . The one or more non-transitory, computer-readable storage media of claim 17 , wherein the instructions further cause the one or more processors to perform operations comprising:

determining, for the client device, that a first original performance score is below a score threshold; and

in response to determining that the first original performance score is below the score threshold, adjusting a ratio of data to be assigned to a training dataset to be higher to enlarge the training dataset and to reduce the validation dataset.

19 . The one or more non-transitory, computer-readable storage media of claim 13 , wherein the instructions cause the one or more processors to transmit a request to the client device to train the machine learning model, wherein the request comprises a ratio of training data to be used in the training dataset and the validation dataset.

20 . The one or more non-transitory, computer-readable storage media of claim 13 , wherein the instructions cause the one or more processors to perform operations comprising:

storing, for each client device of a plurality of client devices, a corresponding set of original performance scores and corresponding set of aggregated model parameters; and

in response to determining that the machine learning model corresponding to the client device is associated with the performance score lower than the threshold, selecting a first set of aggregated model parameters for the client device, wherein the first set of aggregated model parameters is associated with a highest original performance score.