Computer-readable recording medium storing evaluation program, evaluation method, and information processing device
A recording medium storing a program causing a computer to execute: acquiring, for each model, an attack result of an estimation attack, the acquiring for a respective model being performed by using the respective model and a plurality of data, each model being a model trained by using synthetic data that simulates training data, an amount of the synthetic data for each model being different from each other; specifying, based on the attack result for each model, specific data from among the plurality of data; performing the estimation attack by using the specific data for a specific model of which an amount of the synthetic data is between the amounts of the synthetic data used for any two models; and evaluating, based on an attack result for the specific data and the attack result for each model, a resistance to the estimation attack for the specific model.
1 . A non-transitory computer-readable recording medium storing an evaluation program for causing a computer to execute processing comprising:
acquiring, for each of a plurality of machine learning models, an attack result of a membership inference attack performed on the each of plurality of machine learning models, the acquiring of the attack result for a respective machine learning model being performed by using the respective machine learning model and a plurality of pieces of data to be attacked by the membership inference attack, each of the plurality of machine learning models being a machine learning model trained by using synthetic data that simulates training data, an amount of the synthetic data used for each of the plurality of machine learning models being different from each other;
specifying, based on the acquired attack result for each of the plurality of machine learning models, specific data from among the plurality of pieces of data;
performing the membership inference attack by using the specific data for a specific machine learning model, the specific machine learning model being a machine learning model for which an amount of the synthetic data is between amounts of the synthetic data used for training of any two machine learning models of the plurality of learning models; and
evaluating, based on an attack result for the specific data and the attack result acquired for each of the plurality of machine learning models, a resistance to the membership inference attack for the specific machine learning model.
2 . The non-transitory computer-readable recording medium according to claim 1 , wherein
the specifying
specifies data with a different attack result of the attack results from among the plurality of pieces of data as the specific data.
3 . The non-transitory computer-readable recording medium according to claim 2 , wherein
the evaluating
selects each attack result regarding data other than the specific data,
generates an attack result for the specific machine learning model by using each selected attack result and the attack result for the specific data, and
calculates a rate of attack failures in the attack result for the specific machine learning model as the resistance to the membership inference attack for the specific machine learning model.
4 . The non-transitory computer-readable recording medium according to claim 1 , for causing the computer to execute processing further comprising:
generating a plurality of machine learning models by using a plurality of training datasets of which the synthetic data amounts are different from each other;
performing the membership inference attack on a first machine learning model generated by using a training dataset that does not include the synthetic data and generating a first attack result that indicates whether or not it is possible to estimate each of the plurality of pieces of data; and
performing the membership inference attack on a second machine learning model generated by using a training dataset that includes the most pieces of synthetic data and generating a second attack result that indicates whether or not it is possible to estimate each of the plurality of pieces of data.
5 . The non-transitory computer-readable recording medium according to claim 4 , wherein
the acquiring
acquires the first attack result and the second attack result,
the specifying compares the first attack result and the second attack result and specifies the specific data with a different attack result from among the plurality of pieces of data,
the performing
performs the membership inference attack by using the specific data on a third machine learning model of which an amount of the synthetic data is an intermediate amount between the first machine learning model and the second machine learning model, and
the evaluating
evaluates a resistance to the membership inference attack for the third machine learning model on the basis of the first attack result or the second attack result and the attack result for the specific data.
6 . The non-transitory computer-readable recording medium according to claim 5 , wherein
the acquiring
sequentially specifies an unevaluated machine learning model until evaluation of the resistance to the membership inference attack for each of the plurality of machine learning models is completed,
specifies a machine learning model of which an amount of the synthetic data is less than the unevaluated machine learning model and a machine learning model of which an amount of the synthetic data is more than the unevaluated machine learning model from among the plurality of machine learning models, and
acquires the attack results of the two specified machine learning models, the specifying
compares the attack results of the two machine learning models and specifies the specific data,
the performing
performs the membership inference attack by using the specific data on the unevaluated machine learning model, and
the evaluating
evaluates a resistance to the membership inference attack for the unevaluated machine learning model on the basis of the attack results of the two machine learning models and the attack result for the specific data.
7 . The non-transitory computer-readable recording medium according to claim 6 , wherein
the acquiring, the specifying, the performing, and the evaluating recursively repeat processing until evaluation of a resistance to the membership inference attack for each of the plurality of machine learning models is completed.
8 . A computer-implemented evaluation method comprising:
acquiring, for each of a plurality of machine learning models, an attack result of a membership inference attack performed on the each of plurality of machine learning models, the acquiring of the attack result for a respective machine learning model being performed by using the respective machine learning model and a plurality of pieces of data to be attacked by the membership inference attack, each of the plurality of machine learning models being a machine learning model trained by using synthetic data that simulates training data, an amount of the synthetic data used for each of the plurality of machine learning models being different from each other;
specifying, based on the acquired attack result for each of the plurality of machine learning models, specific data from among the plurality of pieces of data;
performing the membership inference attack by using the specific data for a specific machine learning model, the specific machine learning model being a machine learning model for which an amount of the synthetic data is between amounts of the synthetic data used for training of any two machine learning models of the plurality of learning models; and
evaluating, based on an attack result for the specific data and the attack result acquired for each of the plurality of machine learning models, a resistance to the membership inference attack for the specific machine learning model.
9 . An information processing apparatus comprising:
a memory; and
a processor coupled to the memory, the processor being configured to perform processing, the processing including:
acquiring, for each of a plurality of machine learning models, an attack result of a membership inference attack performed on the each of plurality of machine learning models, the acquiring of the attack result for a respective machine learning model being performed by using the respective machine learning model and a plurality of pieces of data to be attacked by the membership inference attack, each of the plurality of machine learning models being a machine learning model trained by using synthetic data that simulates training data, an amount of the synthetic data used for each of the plurality of machine learning models being different from each other;
specifying, based on the acquired attack result for each of the plurality of machine learning models, specific data from among the plurality of pieces of data;
performing the membership inference attack by using the specific data for a specific machine learning model, the specific machine learning model being a machine learning model for which an amount of the synthetic data is between amounts of the synthetic data used for training of any two machine learning models of the plurality of learning models; and
evaluating, based on an attack result for the specific data and the attack result acquired for each of the plurality of machine learning models, a resistance to the membership inference attack for the specific machine learning model.