Edge data distribution cliques
Techniques described herein relate to a method for model updating based on maximal cliques. The method may include transmitting, by a model coordinator, a probability distribution request signal to a plurality of edge nodes; receiving, by the model coordinator, a separate feature probability distribution from each of the plurality of edge nodes; executing, by the model coordinator, a maximal clique identification algorithm using the feature probability distributions to obtain a plurality of maximal cliques; selecting, by the model coordinator, a representative edge node from each of the plurality of maximal cliques to obtain a plurality of representative edge nodes; transmitting, by the model coordinator, a feature data request signal to each of the plurality of representative edge nodes; receiving, by the model coordinator, feature data from each of the plurality of representative edge nodes; and performing machine learning (ML) model training using a first portion of the feature data.
1 . A method for model updating based on maximal cliques, the method comprising:
transmitting, by a model coordinator, a probability distribution request signal to a plurality of edge nodes, wherein the plurality of edge nodes obtain telemetry feature data related to one or more features of an edge device specified in the probability distribution request signal and wherein the one or more features are a subset of features from a previous probability distribution request signal from a previous machine learning (ML) model training cycle;
receiving, by the model coordinator, a separate feature probability distribution from each of the plurality of edge nodes, wherein the separate feature probability distribution is calculated using the telemetry feature data;
executing, by the model coordinator, a maximal clique identification algorithm using the feature probability distributions to obtain a first plurality of maximal cliques, wherein:
a maximal clique of the first plurality of maximal cliques is a set of one or more edge nodes that have similar probability distributions for the one or more features, and the maximal clique identification algorithm comprises:
calculating a divergence value, based on the feature probability distributions, between two edge nodes of the plurality of edge nodes using a bounded symmetric divergence metric,
comparing the divergence value with a divergence value threshold to obtain a result, and
determining that the two edge nodes are in a maximal clique based on the result;
selecting, by the model coordinator, a representative edge node from each of the first plurality of maximal cliques to obtain a plurality of representative edge nodes, wherein a technique for selecting the representative edge node is a round robin scheme;
transmitting, by the model coordinator, a feature data request signal to each of the plurality of representative edge nodes, wherein the feature data request signal includes a type of feature requested based on the previous ML model training cycle;
receiving, by the model coordinator, feature data for the one or more features from each of the plurality of representative edge nodes;
performing ML model training on an ML model using a first portion of the feature data to obtain a trained ML model;
performing ML model validation, on the trained ML model, using a second portion of the feature data to obtain a current validation metric, wherein the current validation metric is stored in metadata along with other information relevant to a future ML model training cycle;
comparing the current validation metric to a previous validation metric from the previous ML model training cycle;
updating the divergence value threshold for the future ML model training cycle based on the comparison of the current validation metric and the previous validation metric to obtain an updated divergence value threshold, wherein the divergence value threshold is raised because the trained ML model performed better to the previous ML model training cycle;
identifying a set of important features for the future ML model training cycle using a random forest algorithm that inherently ranks the one or more features during training to further refine a set of features for the future ML model training cycle; and
after the identifying, initiating further training of the trained ML model using the set of a set of important features for the future ML model training cycle, and the updated divergence value threshold, wherein the further training includes a second plurality of maximal cliques with a first amount of maximal cliques less than the first plurality of maximal cliques reducing an amount of data to train the ML model during the future ML model training cycle.
2 . The method of claim 1 , wherein a maximal clique of the first plurality of maximal cliques comprises a portion of the plurality of edge nodes.
3 . The method of claim 1 , wherein each of the plurality of edge nodes prepares and transmits the separate feature probability distribution to the model coordinator in response to receiving the probability distribution request signal.
4 . A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for model updating based on maximal cliques, the method comprising:
transmitting, by a model coordinator, a probability distribution request signal to a plurality of edge nodes, wherein the plurality of edge nodes obtain telemetry feature data related to one or more features of an edge device specified in the probability distribution request signal and wherein the one or more features are a subset of features from a previous probability distribution request signal from a previous machine learning (ML) model training cycle;
receiving, by the model coordinator, a separate feature probability distribution from each of the plurality of edge nodes, wherein the separate feature probability distribution is calculated using the telemetry feature data;
executing, by the model coordinator, a maximal clique identification algorithm using the feature probability distributions to obtain a first plurality of maximal cliques, wherein:
a maximal clique of the first plurality of maximal cliques is a set of one or more edge nodes that have similar probability distributions for the one or more features, and
the maximal clique identification algorithm comprises:
calculating a divergence value, based on the feature probability distributions, between two edge nodes of the plurality of edge nodes using a bounded symmetric divergence metric,
comparing the divergence value with a divergence value threshold to obtain a result, and
determining that the two edge nodes are in a maximal clique based on the result;
selecting, by the model coordinator, a representative edge node from each of the first plurality of maximal cliques to obtain a plurality of representative edge nodes, wherein a technique for selecting the representative edge node is a round robin scheme;
transmitting, by the model coordinator, a feature data request signal to each of the plurality of representative edge nodes, wherein the feature data request signal includes a type of feature requested based on the previous ML model training cycle;
receiving, by the model coordinator, feature data for the one or more features from each of the plurality of representative edge nodes;
performing ML model training on an ML model using a first portion of the feature data to obtain a trained ML model;
performing ML model validation, on the trained ML model, using a second portion of the feature data to obtain a current validation metric, wherein the current validation metric is stored in metadata along with other information relevant to a future ML model training cycle;
comparing the current validation metric to a previous validation metric from the previous ML model training cycle;
updating the divergence value threshold for the future ML model training cycle based on the comparison of the current validation metric and the previous validation metric to obtain an updated divergence value threshold, wherein the divergence value threshold is raised because the trained ML model performed better to the previous ML model training cycle;
identifying a set of important features for the future ML model training cycle using a random forest algorithm that inherently ranks the one or more features during training to further refine a set of features for the future ML model training cycle; and
after the identifying, initiating further training of the trained ML model using the set of a set of important features for the future ML model training cycle, and the updated divergence value threshold, wherein the further training includes a second plurality of maximal cliques with a first amount of maximal cliques less than the first plurality of maximal cliques reducing an amount of data to train the ML model during the future ML model training cycle.
5 . The non-transitory computer readable medium of claim 4 , wherein a maximal clique of the first plurality of maximal cliques comprises a portion of the plurality of edge nodes.
6 . The non-transitory computer readable medium of claim 4 , wherein each of the plurality of edge nodes prepares and transmits the separate feature probability distribution of the model coordinator in response to receiving the probability distribution request signal.
7 . A system for model updating based on maximal cliques, the system comprising:
a model coordinator, executing on a processor comprising circuitry, and configured to:
transmit, by a model coordinator, a probability distribution request signal to a plurality of edge nodes, wherein the plurality of edge nodes obtain telemetry feature data related to one or more features of an edge device specified in the probability distribution request signal and wherein the one or more features are a subset of features from a previous probability distribution request signal from a previous machine learning (ML) model training cycle;
receive, by the model coordinator, a separate feature probability distribution from each of the plurality of edge nodes, wherein the separate feature probability distribution is calculated using the telemetry feature data;
execute, by the model coordinator, a maximal clique identification algorithm using the feature probability distributions to obtain a first plurality of maximal cliques, wherein:
a maximal clique of the first plurality of maximal cliques is a set of one or more edge nodes that have similar probability distributions for the one or more features, and
the maximal clique identification algorithm comprises:
calculating a divergence value, based on the feature probability distributions, between two edge nodes of the plurality of edge nodes using a bounded symmetric divergence metric,
comparing the divergence value with a divergence value threshold to obtain a result, and
determining that the two edge nodes are in a maximal clique based on the result;
select, by the model coordinator, a representative edge node from each of the first plurality of maximal cliques to obtain a plurality of representative edge nodes, wherein a technique for selecting the representative edge node is a round robin scheme;
transmit, by the model coordinator, a feature data request signal to each of the plurality of representative edge nodes, wherein the feature data request signal includes a type of feature requested based on the previous ML model training cycle;
receive, by the model coordinator, feature data for the one or more features from each of the plurality of representative edge nodes;
perform ML model training on an ML model using a first portion of the feature data to obtain a trained ML model;
perform ML model validation, on the trained ML model, using a second portion of the feature data to obtain a current validation metric, wherein the current validation metric is stored in metadata along with other information relevant to a future ML model training cycle;
compare the current validation metric to a previous validation metric from the previous ML model training cycle;
update the divergence value threshold for the future ML model training cycle based on the comparison of the current validation metric and the previous validation metric to obtain an updated divergence value threshold, wherein the divergence value threshold is raised because the trained ML model performed better to the previous ML model training cycle;
identify a set of important features for the future ML model training cycle using a random forest algorithm that inherently ranks the one or more features during training to further refine a set of features for the future ML model training cycle; and
after the identifying, initiating further training of the trained ML model using the set of a set of important features for the future ML model training cycle, and the updated divergence value threshold, wherein the further training includes a second plurality of maximal cliques with a first amount of maximal cliques less than the first plurality of maximal cliques reducing an amount of data to train the ML model during the future ML model training cycle.
8 . The system of claim 7 , wherein a maximal clique of the first plurality of maximal cliques comprises a portion of the plurality of edge nodes.
9 . The system of claim 7 , wherein each of the plurality of edge nodes prepares and transmits the separate feature probability distribution to the model coordinator in response to receiving the probability distribution request signal.