Learning model management system, learning model management method, and recording medium
A learning model management system executes a provisional evaluation when the number of feedback data is equal to or less than a threshold of a definite evaluation but is more than a threshold of a provisional evaluation. In this provisional evaluation, prediction accuracy of the learning model that is in operation is evaluated a plurality of times, and whether or not the prediction accuracy of the learning model that is in operation is in a deterioration trend is determined based on the change tendency of the evaluation results. If it is determined that the prediction accuracy of the learning model that is in operation is in a deterioration trend, the learning model management system provides a notification about the deterioration trend of the prediction accuracy to a manager to cause him/her to increase the number of feedback data up to a number that enables the definite evaluation.
1. A learning model management system that uses a learning model to estimate information to be provided to a user based on user information of the user and to evaluate performance of the learning model, comprising:
a memory storing instructions; and
a processor executing the instructions causing the learning model management system to:
receive an input for specifying a number of feedback data required for performing a definite performance evaluation of the learning model;
perform the definite performance evaluation of the learning model when a number of feedback data from a providing destination exceeds a first threshold which is the specified number of feedback data;
perform relearning of the learning model when a deterioration of the performance of the learning model is detected as a result of the definite performance evaluation of the learning model;
perform a provisional performance evaluation of the learning model when the number of feedback data from the providing destination does not exceed the first threshold and exceeds a second threshold smaller than the first threshold based on the specified number of feedback data; and
provide a notification corresponding to the deterioration of the performance when the deterioration of the performance of the learning model has been detected based on result from the provisional performance evaluation performed a plurality of times,
wherein the notification is for requesting an administrator of the learning model management system to collect the first threshold number of feedback data from the providing destination.
2. The learning model management system according to claim 1 , wherein whether or not the performance of the learning model deteriorates is determined based on a change tendency of the performance of the learning model obtained based on the result from the provisional performance evaluation performed the plurality of times.
3. The learning model management system according to claim 1 , wherein the notification includes information for specifying a target user recommended as a feedback data collection target based on one or more types of user attributes defined in advance.
4. The learning model management system according to claim 3 , wherein the notification includes a recommendation list created by extracting the information related to the target user based on the user attributes from a user table stored in advance.
5. The learning model management system according to claim 1 , wherein the relearning is executed by using at least a part of data included in the feedback data from the providing destination to serve as learning data.
6. A learning model management method that uses a learning model to estimate information to be provided to a user based on user information of the user and to evaluate performance of the learning model, the method comprising:
receiving an input for specifying a number of feedback data required for performing a definite performance evaluation of the learning model;
performing the definite performance evaluation of the learning model when a number of feedback data from a providing destination exceeds a first threshold which is the specified number of feedback data;
performing relearning of the learning model when a deterioration of the performance of the learning model is detected as a result of the definite performance evaluation of the learning model;
performing a provisional performance evaluation of the learning model when the number of feedback data from the providing destination does not exceed the first threshold and exceeds a second threshold smaller than the first threshold based on the specified number of feedback data; and
providing a notification corresponding to the deterioration of the performance when the deterioration of the performance of the learning model has been detected based on result from the provisional performance evaluation performed a plurality of times,
wherein the notification is for requesting an administrator of the learning model management method to collect the first threshold number of feedback data from the providing destination.
7. A non-transitory storage medium on which is stored a computer program for a learning model management system that uses a learning model to estimate information to be provided to a user based on the user information of the user and to evaluate performance of the learning model, the computer program causing a processor of the learning model management system to execute:
receiving an input for specifying a number of feedback data for performing a definite performance evaluation of the learning model;
performing the definite performance evaluation of the learning model when a number of feedback data from a providing destination exceeds a first threshold which is the specified number of feedback data;
performing relearning of the learning model when a deterioration of the performance of the learning model is detected as a result of the definite performance evaluation of the learning model;
performing a provisional performance evaluation of the learning model when the number of feedback data from the providing destination does not exceed the first threshold and exceeds a second threshold smaller than the first threshold based on the specified number of feedback data; and
providing a notification corresponding to the deterioration of the performance when the deterioration of the performance of the learning model has been detected based on the result from the provisional performance evaluation performed a plurality of times,
wherein the notification is for requesting an administrator of the learning model system to collect the first threshold number of feedback data from the providing destination.