Computer-implemented detection method for detecting accuracy degradation of machine learning model, non-transitory computer-readable recording medium, and computing system
A computing system calculates, by using an inspector model, whether or not the plurality of pieces of training data are located in a vicinity of the decision boundary, acquires a first proportion of the training data, calculates, by using the inspector model, whether or not a plurality of pieces of operation data associated with one of correct answer labels out of the plurality of correct answer labels are located in a vicinity of the decision boundary, and acquires a second proportion of the operation data located in the vicinity of the decision boundary out of all of the pieces of operation data and detects, based on the first proportion and the second proportion, a change in the output result of the machine learning model caused by a temporal change in a tendency of the operation data.
1 . A computer-implemented detection method comprising:
inputting, to a machine learning model classifying high-dimensional image data, a plurality of pieces of training data configured with one of correct answer labels out of three or more types of correct answer labels, and training the machine learning model such that output results respectively match the correct answer labels configured for the input training data;
creating an inspector model configured to calculate a distance to operation data from a decision boundary for classifying a feature space of data into a plurality of application areas, the inspector model having learned the decision boundary by performing machine learning to allow the output results obtained by inputting the plurality of pieces of training data to the trained machine learning model to respectively match output results obtained by inputting the plurality of pieces of training data to the inspector model;
determining during the training of the inspector model, by using the inspector model, whether or not the plurality of pieces of training data are located in an area in which a distance from the decision boundary is less than or equal to a threshold, and acquiring, from the training data as a whole, a first proportion of the training data located in the area in which the distance from the decision boundary is less than or equal to the threshold to store the first proportion of the training data;
inputting to the inspector model, during an operation of the machine learning model and the inspector model, a plurality of pieces of the operation data that are classified by the machine learning model and that are configured with one of the correct answer labels out of three or more types of correct answer labels so as to calculate using the inspector model whether or not the plurality of pieces of the operation data are located in the area in which the distance from the decision boundary is less than or equal to the threshold, and acquiring, from the training data as a whole, a second proportion of the operation data located in the area in which the distance from the decision boundary is less than or equal to the threshold; and
in response to identifying that the second proportion acquired during the operation is increased or decreased from the first proportion stored during the training detecting degradation in accuracy of the classification of the high-dimensional image data performed by the machine learning model.
2 . The computer-implemented detection method according to claim 1 , wherein the creating includes creating a plurality of the inspector models obtained by training the decision boundary that classifies the feature space of the data into one of the application areas and the other application areas.
3 . The computer-implemented detection method according to claim 2 , wherein the acquiring the first proportion includes acquiring the first proportion for each decision boundary of the plurality of inspector models, and the acquiring the second proportion includes acquiring the second proportion for each decision boundary of the plurality of inspector models.
4 . The computer-implemented detection method according to claim 3 , wherein the detecting includes detecting data corresponding to a cause of the change in the output result of the machine learning model based on the first proportion for each decision boundary in the plurality of inspector models and the second proportion for each decision boundary in the plurality of inspector models.
5 . A non-transitory computer-readable recording medium having stored therein a detection program executable by one or more computers, the detection program comprising:
instructions for inputting, to a machine learning model classifying high-dimensional image data, a plurality of pieces of training data configured with one of correct answer labels out of three or more types of correct answer labels, and training the machine learning model such that output results respectively match the correct answer labels configured for the input training data;
instructions for creating an inspector model configured to calculate a distance to operation data from a decision boundary for classifying a feature space of data into a plurality of application areas, the inspector model having learned the decision boundary by performing machine learning to allow the output results obtained by inputting the plurality of pieces of training data to the trained machine learning model to respectively match output results obtained by inputting the plurality of pieces of training data to the inspector model;
instructions for determining during the training of the inspector model, by using the inspector model, whether or not the plurality of pieces of training data are located in an area in which a distance from the decision boundary is less than or equal to a threshold, and acquiring, from the training data as a whole, a first proportion of the training data located in the area in which the distance from the decision boundary is less than or equal to the threshold to store the first proportion of the training data;
instructions for inputting to the inspector model, during an operation of the machine learning model and the inspector model, a plurality of pieces of the operation data that are classified by the machine learning model and that are configured with one of the correct answer labels out of three or more types of correct answer labels so as to calculate using the inspector model whether or not the plurality of pieces of the operation data are located in the area in which the distance from the decision boundary is less than or equal to the threshold, and acquiring, from the training data as a whole, a second proportion of the operation data located in the area in which the distance from the decision boundary is less than or equal to the threshold; and
instructions in response to identifying that the second proportion acquired during the operation is increased or decreased from the first proportion stored during the training, detecting degradation in accuracy of the classification of the high-dimensional image data performed by the machine learning model.
6 . The non-transitory computer-readable recording medium according to claim 5 , wherein the creating includes creating a plurality of the inspector models obtained by training the decision boundary that classifies the feature space of the data into one of the application areas and the other application areas.
7 . The non-transitory computer-readable recording medium according to claim 6 , wherein the acquiring the first proportion includes acquiring the first proportion for each decision boundary of the plurality of inspector models, and
the acquiring the second proportion includes acquiring the second proportion for each decision boundary of the plurality of inspector models.
8 . The non-transitory computer-readable recording medium according to claim 7 , wherein the detecting includes detecting data corresponding to a cause of the change in the output result of the machine learning model based on the first proportion for each decision boundary in the plurality of inspector models and the second proportion for each decision boundary in the plurality of inspector models.
9 . A computing system comprising:
one or more memories; and
one or more processors coupled to the one or more memories, the one or more processors configured to
input, to a machine learning mode classifying high-dimensional image data 1 , a plurality of pieces of training data configured with one of correct answer labels out of three or more types of correct answer labels, and train the machine learning model such that output results respectively match the correct answer labels configured for the input training data,
create an inspector model configured to calculate a distance to operation data from a decision boundary for classifying a feature space of data into a plurality of application areas, the inspector model having learned the decision boundary by performing machine learning to allow the output results obtained by inputting the plurality of pieces of training data to the trained machine learning model to respectively match output results obtained by inputting the plurality of pieces of training data to the inspector model,
determine during the training of the inspector model, by using the inspector model, whether or not the plurality of pieces of training data are located in an area in which a distance from the decision boundary is less than or equal to a threshold, and acquire, from the training data as a whole, a first proportion of the training data located in the area in which the distance from the decision boundary is less than or equal to the threshold to store the first proportion of the training data,
input to the inspector model, during an operation of the machine learning model and the inspector model, a plurality of pieces of the operation data that are classified by the machine learning model and that are configured with one of the correct answer labels out of three or more types of correct answer labels so as to calculate using the inspector model whether or not the plurality of pieces of the operation data are located in the area in which the distance from the decision boundary is less than or equal to the threshold, and acquire, from the training data as a whole, a second proportion of the operation data located in the area in which the distance from the decision boundary is less than or equal to the threshold, and
in response to identifying that the second proportion acquired during the operation is increased or decreased from the first proportion stored during the training, detect degradation in accuracy of the classification of the high-dimensional image data performed by the machine learning model.
10 . The computing system according to claim 9 , the processor further configured to create a plurality of the inspector models obtained by training the decision boundary that classifies the feature space of the data into one of the application areas and the other application areas.
11 . The computing system according to claim 10 , the processor further configured to acquire the first proportion for each decision boundary of the plurality of inspector models and acquire the second proportion for each decision boundary of the plurality of inspector models.
12 . The computing system according to claim 11 , the processor further configured to detect data corresponding to a cause of the change in the output result of the machine learning model based on the first proportion for each decision boundary in the plurality of inspector models and the second proportion for each decision boundary in the plurality of inspector models.