System and method for preventing introduction of poisoned training data to artificial intelligence models
Methods and systems for managing artificial intelligence (AI) models are disclosed. To manage AI models, an instance of an AI model may not be re-trained using training data determined to be too similar to previously used training data. By doing so, malicious attacks intending to shift the AI model in a particular direction using poisoned training data may be prevented. To do so, a candidate training data set may be analyzed prior to performing re-training of an instance of an AI model using the candidate training data set. The analysis may result in a score. If the score exceeds a score threshold, the candidate training data set may be considered to contain poisoned training data. If the score does not exceed the score threshold, the candidate training data set may be accepted as usable to train an instance of the AI model.
1 . A method of managing an artificial intelligence (AI) model, the method comprising:
obtaining a candidate training data set usable to update an instance of the AI model;
prior to updating the instance of the AI model using the candidate training data set:
identifying a historical training data set, the historical training data set being obtained prior to the candidate training data set and the historical training data set already having been used to train the instance of the AI model;
performing an analysis of the candidate training data set and the historical training data set to obtain a score reflecting a likelihood that the candidate training data set comprises poisoned training data, wherein a higher score indicates a higher likelihood that the candidate training data set comprises poisoned training data, wherein performing the analysis comprises:
identifying a first data value of the candidate training data set;
obtaining a similarity measure between the first data value and each data value of the historical training data set to obtain a set of similarity measures;
making a first determination regarding whether each similarity measure of the set of similarity measures exceeds a similarity measure threshold;
in a first instance of the first determination in which at least a portion of the set of similarity measures exceeds the similarity measure threshold, increasing the score; and
in a second instance of the first determination in which at least a portion of the set of similarity measures does not exceed the similarity measure threshold, decreasing the score;
in response to the score exceeding a score threshold:
removing the candidate training data set from consideration as training data for the AI model;
treating the candidate training data set as being part of a malicious attack;
discarding the candidate training data set;
identifying a data source of the candidate training data set; and
treating the data source of the candidate training data set as a potentially malicious data source.
2 . The method of claim 1 , wherein performing the analysis comprises:
performing a cluster analysis of the historical training data set to obtain a set of clusters;
identifying a first data value of the candidate training data set;
making a second determination regarding whether the first data value falls within the set of clusters;
in a first instance of the second determination in which the first data value falls within the set of clusters:
increasing the score; and
in a second instance of the second determination in which the first data value does not fall within the set of clusters:
decreasing the score; and
approving the first data value for AI model training purposes.
3 . The method of claim 2 , wherein making the second determination comprises:
for each of the clusters of the set of clusters:
making a comparison between the first data value and a bounding area of a respective cluster to determine whether the first data value falls within the respective cluster; and
in an instance of the comparison in which the first data value falls within the respective cluster, concluding that the first data value falls within the set of clusters.
4 . The method of claim 3 , wherein the bounding area is defined by a point and a radius that extends about the point thereby establishing the bounding area.
5 . The method of claim 1 , wherein the similarity measure comprises one selected from a group consisting of:
a Euclidean distance;
a cosine similarity; and
a Manhattan distance.
6 . The method of claim 1 , further comprising:
prior to obtaining the candidate training data set, making an identification that a re-training condition is met for the AI model.
7 . A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing an artificial intelligence (AI) model, the operations comprising:
obtaining a candidate training data set usable to update an instance of the AI model;
prior to updating the instance of the AI model using the candidate training data set:
identifying a historical training data set, the historical training data set being obtained prior to the candidate training data set and the historical training data set already having been used to train the instance of the AI model;
performing an analysis of the candidate training data set and the historical training data set to obtain a score reflecting a likelihood that the candidate training data set comprises poisoned training data, wherein a higher score indicates a higher likelihood that the candidate training data set comprises poisoned training data, wherein performing the analysis comprises:
identifying a first data value of the candidate training data set;
obtaining a similarity measure between the first data value and each data value of the historical training data set to obtain a set of similarity measures;
making a first determination regarding whether each similarity measure of the set of similarity measures exceeds a similarity measure threshold;
in a first instance of the first determination in which at least a portion of the set of similarity measures exceeds the similarity measure threshold, increasing the score; and
in a second instance of the first determination in which at least a portion of the set of similarity measures does not exceed the similarity measure threshold, decreasing the score;
in response to the score exceeding a score threshold:
removing the candidate training data set from consideration as training data for the AI model;
treating the candidate training data set as being part of a malicious attack;
discarding the candidate training data set;
identifying a data source of the candidate training data set; and
treating the data source of the candidate training data set as a potentially malicious data source.
8 . The non-transitory machine-readable medium of claim 7 , wherein performing the analysis comprises:
performing a cluster analysis of the historical training data set to obtain a set of clusters;
identifying a first data value of the candidate training data set;
making a second determination regarding whether the first data value falls within the set of clusters;
in a first instance of the second determination in which the first data value falls within the set of clusters:
increasing the score; and
in a second instance of the second determination in which the first data value does not fall within the set of clusters:
decreasing the score; and
approving the first data value for AI model training purposes.
9 . The non-transitory machine-readable medium of claim 8 , wherein making the second determination comprises:
for each of the clusters of the set of clusters:
making a comparison between the first data value and a bounding area of a respective cluster to determine whether the first data value falls within the respective cluster; and
in an instance of the comparison in which the first data value falls within the respective cluster, concluding that the first data value falls within the set of clusters.
10 . The non-transitory machine-readable medium of claim 9 , wherein the bounding area is defined by a point and a radius that extends about the point thereby establishing the bounding area.
11 . The non-transitory machine-readable medium of claim 7 , wherein the similarity measure comprises one selected from a group consisting of:
a Euclidean distance;
a cosine similarity; and
a Manhattan distance.
12 . The non-transitory machine-readable medium of claim 7 , wherein the operations further comprise:
prior to obtaining the candidate training data set, making an identification that a re-training condition is met for the AI model.
13 . The non-transitory machine-readable medium of claim 12 , wherein the candidate training data set is obtained in response to the identification.
14 . A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing an artificial intelligence (AI) model, the operations comprising:
obtaining a candidate training data set usable to update an instance of the AI model;
prior to updating the instance of the AI model using the candidate training data set:
identifying a historical training data set, the historical training data set being obtained prior to the candidate training data set and the historical training data set already having been used to train the instance of the AI model;
performing an analysis of the candidate training data set and the historical training data set to obtain a score reflecting a likelihood that the candidate training data set comprises poisoned training data, wherein a higher score indicates a higher likelihood that the candidate training data set comprises poisoned training data, wherein performing the analysis comprises:
identifying a first data value of the candidate training data set;
obtaining a similarity measure between the first data value and each data value of the historical training data set to obtain a set of similarity measures;
making a first determination regarding whether each similarity measure of the set of similarity measures exceeds a similarity measure threshold;
in a first instance of the first determination in which at least a portion of the set of similarity measures exceeds the similarity measure threshold, increasing the score; and
in a second instance of the first determination in which at least a portion of the set of similarity measures does not exceed the similarity measure threshold, decreasing the score;
in response to the score exceeding a score threshold:
removing the candidate training data set from consideration as training data for the AI model;
treating the candidate training data set as being part of a malicious attack;
discarding the candidate training data set;
identifying a data source of the candidate training data set; and
treating the data source of the candidate training data set as a potentially malicious data source.
15 . The data processing system of claim 14 , wherein performing the analysis comprises:
performing a cluster analysis of the historical training data set to obtain a set of clusters;
identifying a first data value of the candidate training data set;
making a second determination regarding whether the first data value falls within the set of clusters;
in a first instance of the second determination in which the first data value falls within the set of clusters:
increasing the score; and
in a second instance of the second determination in which the first data value does not fall within the set of clusters:
decreasing the score; and
approving the first data value for AI model training purposes.
16 . The data processing system of claim 15 , wherein making the second determination comprises:
for each of the clusters of the set of clusters:
making a comparison between the first data value and a bounding area of a respective cluster to determine whether the first data value falls within the respective cluster; and
in an instance of the comparison in which the first data value falls within the respective cluster, concluding that the first data value falls within the set of clusters.
17 . The data processing system of claim 16 , wherein the bounding area is defined by a point and a radius that extends about the point thereby establishing the bounding area.
18 . The data processing system of claim 14 , wherein the similarity measure comprises one selected from a group consisting of:
a Euclidean distance;
a cosine similarity; and
a Manhattan distance.
19 . The data processing system of claim 14 , wherein the operations further comprise:
prior to obtaining the candidate training data set, making an identification that a re-training condition is met for the AI model.
20 . The data processing system of claim 19 , wherein the candidate training data set is obtained in response to the identification.