Identifying, or checking integrity of, a machine-learning classification model

1 . A method comprising:

obtaining, by a processor associated with a classification system, a first set that comprises a plurality of candidate samples applicable to a specific machine-learning classification model; and

updating, by the processor, the first set, said updating comprising, for each candidate sample in the plurality of candidate samples, performing a corresponding sequence of one or more update steps, wherein each update step comprises:

generating a second candidate sample based on said candidate sample;

generating, for each of said candidate sample and the second candidate sample, a corresponding classification measure using the specific machine-learning classification model; and

assessing the generated classification measures, wherein said assessing comprises, based on a comparison of the generated classification measures, performing at least one of:

terminating the sequence of one or more update steps; or

setting said candidate sample to be the second candidate sample if the comparison indicates that the second candidate sample is more likely than the said candidate sample to have a corresponding small modification that causes a change in classification result produced using the specific machine-learning classification model; and

generating, by the processor, a test set based on the updated first set, the test set comprising a plurality of test samples corresponding to some or all of the candidate samples in the updated first set, the test set associated with the specific machine-learning classification model, wherein, for each test sample in the test set, there is a corresponding small modification for that test sample that causes a change in the classification result produced for that test sample using the specific machine-learning classification model.

2 . The method of claim 1 , wherein said generating a second candidate sample based on said candidate sample comprises generating the second candidate sample by performing a random change to said candidate sample.

3 . The method of claim 1 , wherein, for each candidate sample, for each update step of a first subsequence of the corresponding sequence of one or more update steps:

for each of said candidate sample and the second candidate sample, the corresponding classification measure is a score generated by:

using the specific machine-learning classification model to generate a corresponding plurality of values, each value indicative of that sample belonging to a corresponding class; and

using a score function to generate the score for that sample based on the corresponding plurality of values, the score indicative of a likelihood that there is a small modification for that sample that causes a change in the classification result produced for that sample using the specific machine-learning classification model; and

assessing the generated classification measures comprises: if the classification measure for the second candidate sample is indicative of a higher likelihood than the classification measure for said candidate sample, setting said candidate sample to be the second candidate sample.

4 . The method of claim 3 , wherein, for each candidate sample, for each update step of a second subsequence of the corresponding sequence of one or more update steps after the first subsequence:

for each of said candidate sample and the second candidate sample, the corresponding classification measure is an identification of the class for that sample generated using the specific machine-learning classification model; and

assessing the generated classification measures comprises:

if the classification measure for the second candidate sample is the same as the classification measure for said candidate sample, terminating the second subsequence;

if the classification measure for the second candidate sample is not the same as the classification measure for said candidate sample:

for each of said candidate sample and the second candidate sample:

using the specific machine-learning classification model to generate a corresponding plurality of values, each value indicative of that sample belonging to a corresponding class; and

using a score function to generate the score for that sample based on the corresponding plurality of values, the score indicative of a likelihood that there is a small modification for that sample that causes a change in the classification result produced for that sample using the specific machine-learning classification model; and

if the score for the second candidate sample is indicative of a higher likelihood than the score for said candidate sample, setting said candidate sample to be the second candidate sample.

5 . The method of claim 3 , wherein each value represents a probability that the corresponding sample belongs to the corresponding class.

6 . The method of claim 5 , wherein for each of the first candidate sample and the second candidate sample, the corresponding plurality of values are normalized to have a predetermined total.

7 . The method of claim 1 , wherein, for each candidate sample, for each update step of a first subsequence of the corresponding sequence of one or more update steps:

for each of said candidate sample and the second candidate sample, the corresponding classification measure is an identification of class for that sample generated using the specific machine-learning classification model;

assessing the generated classification measures comprises:

if the classification measure for the second candidate sample is the same as the classification measure for said candidate sample, terminating the first subsequence if a termination condition is met;

if the classification measure for the second candidate sample is not the same as the classification measure for said candidate sample:

setting said candidate sample to be the second candidate sample; and

reducing the size of the random change to be applied to the candidate sample when generating a second candidate sample at the next update step.

8 . The method of claim 1 , wherein one or more of the test samples in the test set are generated as adversarial examples for the specific machine-learning classification.

9 . The method of claim 1 , further comprising:

using the classification system to generate, for each test sample in a predetermined test set that comprises a plurality of test samples, a corresponding classification result; and

identifying either (i) that the classification system is using the specific machine-learning classification model if, for each test sample in the test set, the corresponding classification result matches a classification result produced for that test sample using the specific machine-learning classification model or (ii) that the classification system is not using the specific machine-learning classification model if there is a test sample in the test set for which the corresponding classification result does not match the classification result produced for that test sample using the specific machine-learning classification model.

10 . A test set generation system:

a processor configured to:

obtain a first set that comprises a plurality of candidate samples applicable to a specific machine-learning classification model of a classification system; and

update the first set by, for each candidate sample in the plurality of candidate samples, performing a corresponding sequence of one or more update steps, wherein each update step comprises:

generating a second candidate sample based on said candidate sample;

generating, for each of said candidate sample and the second candidate sample, a corresponding classification measure using the specific machine-learning classification model; and

assessing the generated classification measures, wherein said assessing comprises, based on a comparison of the generated classification measures, performing at least one of:

terminating the sequence of one or more update steps; or

setting said candidate sample to be the second candidate sample if the comparison indicates that the second candidate sample is more likely than the said candidate sample to have a corresponding small modification that causes a change in classification result produced using the specific machine-learning classification model; and

generate a test set based on the updated first set, the test set comprising a plurality of test samples corresponding to some or all of the candidate samples in the updated first set, the test set associated with the specific machine-learning classification model, wherein, for each test sample in the test set, there is a corresponding small modification for that test sample that causes a change in the classification result produced for that test sample using the specific machine-learning classification model.

11 . The test set generation system of claim 10 , wherein the identification system is further configured to:

use the classification system to generate, for each test sample in a predetermined test set that comprises a plurality of test samples, a corresponding classification result; and

identify either (i) that the classification system is using the specific machine-learning classification model if, for each test sample in the test set, the corresponding classification result matches a classification result produced for that test sample using the specific machine-learning classification model or (ii) that the classification system is not using the specific machine-learning classification model if there is a test sample in the test set for which the corresponding classification result does not match the classification result produced for that test sample using the specific machine-learning classification model.