Classifier confidence as a means for identifying data drift
Embodiments of the present systems and methods may provide improved machine learning performance even though data drift has occurred. For example, a method may comprise providing a machine learning model in a computer system, operating the machine learning model using a first dataset to obtain results of the first dataset, operating the machine learning model using a second dataset to obtain results of the second dataset, performing statistical testing on a confidence distribution of results of the first dataset and of results of the second dataset to determine a difference in a result confidence distribution between the first dataset and of the second dataset, and determining whether data included in the second dataset has data drift relative to the first dataset based on the difference in a result confidence distribution between the first dataset and of the second dataset.
1. A method comprising:
providing a non-binary machine learning classification model in a computer system comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor;
operating, at the computer system, the non-binary machine learning classification model using a first dataset to obtain a plurality of labels of the first dataset;
operating, at the computer system, the non-binary machine learning classification model using a second dataset to obtain a plurality of labels of the second dataset, wherein second data set simulates data drift relative to the first data set by the plurality of labels of the second data set having at least one more label than the plurality of labels of the first data set;
performing, at the computer system, a non-parametric statistical test for identity, over a classifier confidence-per-label distribution for at least one label of the plurality of labels of the first dataset and of the second dataset to determine a difference in a resulting confidence-per-label distribution between the first dataset and of the second dataset;
determining, at the computer system, whether the second dataset has data drift relative to the first dataset based on the difference in the resulting confidence-per-label distribution between the first dataset and of the second dataset; and
training the non-binary machine learning classification model based on the determination of whether data drift has occurred from the first data set to the second data set.
2. The method of claim 1 , wherein the first dataset comprises at least a portion of a training dataset and the second dataset comprises as least a portion of a production dataset.
3. The method of claim 1 , wherein the first dataset comprises a first portion of a production dataset and the second dataset comprises a second portion of the production dataset.
4. A system comprising a processor, memory accessible by the processor, and computer program instructions stored in the memory and executable by the processor to perform:
operating a non-binary machine learning classification model using a first dataset to obtain a plurality of labels of the first dataset;
operating the non-binary machine learning classification model using a second dataset to obtain a plurality of labels of the second dataset, wherein second data set simulates data drift relative to the first data set by the plurality of labels of the second data set having at least one more label than the plurality of labels of the first data set;
performing a non-parametric statistical test for identity, over a classifier confidence-per-label distribution for at least one label of the plurality of labels of the first dataset and of the second dataset to determine a difference in a resulting confidence-per-label distribution between the first dataset and of the second dataset;
determining whether the second dataset has data drift relative to the first dataset based on the difference in the resulting confidence-per-label distribution between the first dataset and of the second dataset; and
training the non-binary machine learning classification model based on the determination of whether data drift has occurred from the first data set to the second data set.
5. The system of claim 4 , wherein the first dataset comprises at least a portion of a training dataset and the second dataset comprises as least a portion of a production dataset.
6. The system of claim 4 , wherein the first dataset comprises a first portion of a production dataset and the second dataset comprises a second portion of the production dataset.
7. A computer program product comprising a non-transitory computer readable storage having program instructions embodied therewith, the program instructions executable by a computer, to cause the computer to perform a method comprising:
operating a non-binary machine learning classification model using a first dataset to obtain a plurality of labels of the first dataset;
operating the non-binary machine learning classification model using a second dataset to obtain a plurality of labels of the second dataset, wherein second data set simulates data drift relative to the first data set by the plurality of labels of the second data set having at least one more label than the plurality of labels of the first data set;
performing a non-parametric statistical test for identity, over a classifier confidence-per-label distribution for at least one label of the plurality of labels of the first dataset and of the second dataset to determine a difference in a resulting confidence-per-label distribution between the first dataset and of the second dataset;
determining whether the second dataset has data drift relative to the first dataset based on the difference in the resulting confidence-per-label distribution between the first dataset and of the second dataset; and
training the non-binary machine learning classification model based on the determination of whether data drift has occurred from the first data set to the second data set.
8. The computer program product of claim 7 , wherein the first dataset comprises at least a portion of a training dataset and the second dataset comprises as least a portion of a production dataset.
9. The computer program product of claim 7 , wherein the first dataset comprises a first portion of a production dataset and the second dataset comprises a second portion of the production dataset.