Fingerprinting data to detect variances
A system and method for characterizing the data used to train a model for machine learning inference. Training data and production data may both be fingerprinted, and the fingerprints may be compared to detect undesirable variances between training and production data. This may allow performance issues relating to differences in the training data set versus the production data set to be more easily identified. Parameters used for characterization can be determined based on the type of training data such as numerical data, image data, or audio data.
1 . A method for fingerprinting model training data by at least one processor, the method comprising the steps of:
creating, by the at least one processor, a training data fingerprint of a machine learning (ML) training data set in response to characterizing the ML training data set using a set of parameters, the set of parameters being selected based on a training data type of the ML training data set;
training, via ML, an ML inference model using the ML training data set to create a trained ML inference model;
ML inferencing on a production data set using the trained ML inference model;
creating, by the at least one processor, a production data fingerprint in response to characterizing the production data set using the set of parameters;
comparing, via the at least one processor, the production data fingerprint and the training data fingerprint to determine if a variance exists between the production data fingerprint and the training data fingerprint, wherein the variance indicates a performance issue of the ML inferencing associated with differences between the production data set and the ML training data set.
2 . The method of claim 1 , wherein characterizing the ML training data set is performed during the training of the ML inference model.
3 . The method of claim 1 , further comprising creating an alert in response to the variance between the training data fingerprint and the production data fingerprint being greater than a predetermined threshold.
4 . The method of claim 3 , wherein the predetermined threshold is determined based on the training data type.
5 . The method of claim 1 , wherein the set of parameters is for a set of numeric data which comprise one or more of the following: minimum value, maximum value, Z-score, mean and median.
6 . The method of claim 1 , wherein the set of parameters is for a set of image data which comprise one or more of the following: RGB histograms, greyscale histograms, and auto decoder feature histograms.
7 . The method of claim 1 , wherein the set of parameters is for a set of audio data which comprise an FFT-generated spectrogram.
8 . The method of claim 7 , wherein the FFT-generated spectrogram is characterized as image data.
9 . A system having at least one processor, the system configured to:
create, by the at least one processor, a training data fingerprint of a machine learning (ML) training data set in response to characterization of the ML training data set using a set of parameters, the set of parameters being selected based on a training data type of the ML training data set;
train, via ML, an ML inference model using the ML training data set to create a trained ML inference model;
ML inference on a production data set using the trained ML inferencing model; inference on a production data set;
create, by the at least one processor, a production data fingerprint in response to characterization of the production data set using the set of parameters;
compare, via the at least one processor, the production data fingerprint and the training data fingerprint to determine if a variance exists between the production data fingerprint and the training data fingerprint, wherein the variance indicates a performance issue of the ML inference associated with differences between the production data set and the ML training data set.
10 . The system of claim 9 , wherein characterization the ML training data set is performed during the training of the ML inference model.
11 . The system of claim 9 , further configured to: create an alert in response to the variance between the training data fingerprint and the production data fingerprint exceeding a predetermined threshold.
12 . The system of claim 11 , wherein the predetermined threshold is determined based on the training data type.
13 . The system of claim 9 , wherein the set of parameters is for a set of numeric data which comprise one or more of the following: minimum value, maximum value, Z-score, mean and median.
14 . The system of claim 9 , wherein the set of parameters is for a set of image data which comprise one or more of the following: RGB histograms, greyscale histograms, and auto decoder feature histograms.
15 . The system of claim 9 , wherein the set of parameters is for a set of audio data which comprise an FFT-generated spectrogram.
16 . A server comprising at least one processor, the server configured to:
create, by the at least one processor, a training data fingerprint of a machine learning (ML) training data set in response to characterization of the ML training data set using a set of parameters, the set of parameters being selected based on a training data type of the ML training data set;
train, via ML, an ML inference model using the ML training data set to create a trained ML inference model;
ML inference on a production data set using the trained ML inferencing model; inference on a production data set;
create, by the at least one processor, a production data fingerprint in response to characterization of the production data set using the set of parameters;
compare, via the at least one processor, the production data fingerprint and the training data fingerprint to determine if a variance exists between the production data fingerprint and the training data fingerprint, wherein the variance indicates a performance issue of the ML inference associated with differences between the production data set and the ML training data set.
17 . The server of claim 16 , further configured to: create an alert in response to the variance between the training data fingerprint and the production data fingerprint exceeding a predetermined threshold.