ML model drift detection using modified GAN
Discussed herein are devices, systems, and methods for machine learning (ML) model drift detection. A method can include receiving machine learning (ML) data defining a number of layers of neurons, a number of neurons per each layer, and weights for each neuron of a deployed ML model, operating the deployed ML model in a modified generative adversarial network (GAN) architecture, while operating the deployed ML model, recording output of a hidden layer of the deployed ML model, determining a metric of the output, and re-deploying the deployed ML model and monitoring whether the re-deployed ML model is suffering from ML model drift based on the metric.
1 . A device comprising:
at least one memory including instructions stored thereon; and
processing circuitry configured to execute the instructions, the instructions, when executed, cause the processing circuitry to perform operations comprising:
receiving machine learning (ML) data defining a number of layers of neurons, a number of neurons per each layer, and weights for each neuron of a deployed ML model;
operating the deployed ML model in a modified generative adversarial network (GAN) architecture, the modified GAN architecture includes fake data, from a generator of an unmodified GAN architecture that includes a discriminator, input to the deployed ML model, the modified GAN architecture includes no feedback loops configured to update weights of neurons of the modified GAN architecture;
while operating the deployed ML model, recording output of a hidden layer of the deployed ML model;
determining a metric of the output; and
re-deploying the ML model and monitoring whether the re-deployed ML model is suffering from ML model drift based on the metric.
2 . The device of claim 1 , wherein the operations further comprise training the deployed ML model based on input used to determine the metric, resulting in an ML model that does not suffer from the ML model drift.
3 . The device of claim 1 , wherein the metric is determined per class that is classified by the deployed ML model.
4 . The device of claim 3 , wherein the metric includes one or more of (i) an average confidence for a given class is below a first specified threshold, (ii) a difference between the average confidence for a given class is more than a threshold less than a prior average confidence for the given class, (iii) a variance for a given class is above a second specified threshold, or (iv) a difference between the variance for a given class is more than a threshold more than a prior variance for the given class.
5 . The device of claim 1 , wherein the modified GAN architecture includes the deployed ML model configured to classify the fake data independent of real data.
6 . A non-transitory machine-readable medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:
receiving machine learning (ML) data defining a number of layers of neurons, a number of neurons per each layer, and weights for each neuron of a deployed ML model;
operating the deployed ML model in a modified generative adversarial network (GAN) architecture, the modified GAN architecture includes fake data, from a generator of an unmodified GAN architecture that includes a discriminator, input to the deployed ML model, the modified GAN architecture includes no feedback loops configured to update weights of neurons of the modified GAN architecture;
while operating the deployed ML model, recording output of a hidden layer of the deployed ML model;
determining a metric of the output; and
re-deploying the deployed ML model and monitoring whether the re-deployed ML model is suffering from ML model drift based on the metric.
7 . The non-transitory machine-readable medium of claim 6 , wherein the operations further comprise training the deployed ML model based on input used to determine the metric, resulting in an ML model that does not suffer from the ML model drift.
8 . The non-transitory machine-readable medium of claim 6 , wherein the metric is determined per class that is classified by the deployed ML model.
9 . The non-transitory machine-readable medium of claim 8 , wherein the metric includes one or more of (i) an average confidence for a given class is below a first specified threshold, (ii) a difference between the average confidence for a given class is more than a threshold less than a prior average confidence for the given class, (iii) a variance for a given class is above a second specified threshold, or (iv) a difference between the variance for a given class is more than a threshold more than a prior variance for the given class.
10 . The non-transitory machine-readable medium of claim 6 , wherein the modified GAN architecture includes the deployed ML model configured to classify the fake data independent of real data.
11 . A method comprising:
receiving machine learning (ML) data defining a number of layers of neurons, a number of neurons per each layer, and weights for each neuron of a deployed ML model;
operating the deployed ML model in a modified generative adversarial network (GAN) architecture, the modified GAN architecture includes fake data, from a generator of an unmodified GAN architecture that includes a discriminator, input to the deployed ML model, the modified GAN architecture includes no feedback loops configured to update weights of neurons of the modified GAN architecture;
while operating the deployed ML model, recording output of a hidden layer of the deployed ML model;
determining a metric of the output; and
re-deploying the deployed ML model and monitoring whether the re-deployed ML model is suffering from ML model drift based on the metric.
12 . The method of claim 11 , further comprising training the deployed ML model based on input used to determine the metric, resulting in an ML model that does not suffer from the ML model drift.
13 . The method of claim 11 , wherein the metric is determined per class that is classified by the deployed ML model.
14 . The method of claim 13 , wherein the metric includes one or more of (i) an average confidence for a given class is below a first specified threshold, (ii) a difference between the average confidence for a given class is more than a threshold less than a prior average confidence for the given class, (iii) a variance for a given class is above a second specified threshold, or (iv) a difference between the variance for a given class is more than a threshold more than a prior variance for the given class.
15 . The method of claim 11 , wherein the modified GAN architecture includes the deployed ML model configured to classify the fake data independent of real data.