Detecting and mitigating poison attacks using data provenance

1. A computer-implemented method for provenance-based defense against poison attacks, the method comprising:

receiving one or more observations from one or more data sources, wherein each observation comprises one or more features for training a final prediction model;

receiving provenance data corresponding to each observation;

determining whether some or all of the observations are poisoned based at least in part on the corresponding provenance data; and

in response to determining some or all of the observations are poisoned, removing the poisoned observation(s) from a final training dataset used to train the final prediction model, and

wherein determining whether each observation is poisoned comprises:

determining a provenance feature for the provenance data corresponding to each of the observations;

grouping observations characterized by a same provenance signature of the determined provenance feature;

generating a filtered training dataset excluding one or more of the groups of observations from the training dataset; and

training a first prediction model corresponding to the final prediction model using the filtered training dataset.

2. The computer-implemented method as recited in claim 1 , further comprising training a second prediction model corresponding to the final prediction model using a complete training dataset comprising all of the observations.

3. The computer-implemented method as recited in claim 2 , further comprising:

comparing a performance of the first prediction model against a performance of the second prediction model; and

in response to determining the performance of the second prediction model exceeds the performance of the first prediction model, determining the one or more groups of observations excluded from the filtered training dataset are poisoned.

4. The computer-implemented method as recited in claim 1 , wherein the provenance data are trusted data.

5. The computer-implemented method as recited in claim 1 , wherein the provenance data are associated with the observations as metadata.

6. The computer-implemented method as recited in claim 1 , wherein the provenance data identify an origin of the observation with which the provenance data are associated.

7. The computer-implemented method as recited in claim 1 , wherein the one or more observations comprise a trusted dataset of data points and an untrusted dataset of data points.

8. A computer-implemented method for provenance-based defense against poison attacks, the method comprising:

receiving one or more observations from one or more data sources, wherein the one or more observations comprise a trusted dataset of data points and an untrusted dataset of data points, and wherein each observation comprises one or more features for training a final prediction model;

receiving provenance data corresponding to each observation;

determining whether some or all of the observations are poisoned based at least in part on the corresponding provenance data; and

in response to determining some or all of the observations are poisoned, removing the poisoned observation(s) from a final training dataset used to train the final prediction model,

determining a threshold performance difference indicative of poisoned observations, the determining comprising:

randomly removing a subset of the untrusted dataset of data points from the untrusted dataset of data points to generate a first calibration dataset;

randomly selecting a subset of the trusted dataset of data points to generate a second calibration dataset;

training a prediction model corresponding to the final prediction model using the first calibration dataset;

training the prediction model using the first and second calibration datasets;

computing a difference in a performance of the prediction model when trained using the first calibration dataset against a performance of the prediction model when trained using the first and second calibration datasets; and

setting the threshold performance difference to a value greater than or equal to the difference in the performance of the prediction model when trained using the first calibration dataset versus the performance of the prediction model when trained using the first and second calibration datasets.

9. The computer-implemented method as recited in claim 1 , wherein the prediction model comprises a supervised machine learning algorithm.

10. The computer-implemented method as recited in claim 1 , further comprising determining a threshold performance difference indicative of poisoned observations.

11. The computer-implemented method as recited in claim 10 , wherein determining the threshold performance difference indicative of poisoned observations comprises:

randomly removing a subset of untrusted dataset of data points from an untrusted dataset of data points to generate a first calibration dataset; and

training a prediction model corresponding to the final prediction model using the first calibration dataset.

12. The computer-implemented method as recited in claim 10 , wherein determining the threshold performance difference indicative of poisoned observations comprises:

randomly selecting a subset of a trusted dataset of data points to generate a second calibration dataset; and

training the prediction model using the second calibration dataset.

13. The computer-implemented method as recited in claim 10 , wherein determining the threshold performance difference indicative of poisoned observations comprises:

computing a difference in a performance of the prediction model when trained using a first calibration dataset against a performance of the prediction model when trained using first calibration dataset and a second calibration dataset; and

setting the threshold performance difference to a value greater than or equal to the difference in the performance of the prediction model when trained using the first calibration dataset versus the performance of the prediction model when trained using the first and second calibration datasets.

14. The computer-implemented method as recited in claim 1 , comprising training the final prediction model.

15. The computer-implemented method as recited in claim 1 , wherein the final prediction model is based on a supervised learning model.

16. The computer-implemented method as recited in claim 1 , wherein the poison attack comprises a targeted attack.

17. The computer-implemented method as recited in claim 16 , wherein the targeted attack comprises a false negative attack.

18. The computer-implemented method as recited in claim 16 , wherein the targeted attack comprises a false positive attack.

19. The computer-implemented method as recited in claim 16 , wherein the targeted attack comprises an accuracy reduction attack.

20. A computer-implemented method for provenance-based defense against poison attacks in a fully untrusted data environment, the method comprising:

receiving a dataset and associated provenance data, wherein the dataset comprises a plurality of untrusted data points and excludes trusted data points, wherein each untrusted data point is associated with one or more provenance signatures of the provenance data;

randomly assigning a first portion of the untrusted data points to a training dataset;

randomly assigning a second portion of the untrusted data points to a full evaluation dataset;

grouping the untrusted data points of the training dataset into a plurality of groups each characterized by a different one of the provenance signatures, wherein untrusted data points of each group are characterized by a same one of the provenance signatures;

for each group of the untrusted data points in the training dataset:

training a supervised learning method using a first training dataset to generate a complete prediction model, wherein the first training dataset includes all of the untrusted data points;

training the supervised learning method using a second training dataset to generate a filtered prediction model, wherein the second training dataset excludes the group of the untrusted data points;

generating a new evaluation dataset by removing, from the full evaluation dataset, any data points sharing a provenance signature with one or more data points of the second training dataset;

applying each of the complete prediction model and the filtered prediction model to the new evaluation dataset;

comparing a performance of applying the complete prediction model to the new evaluation dataset to a performance of applying the filtered prediction model to the new evaluation dataset to determine whether the performance of applying the filtered prediction model to the new evaluation dataset exceeds the performance of applying the complete prediction model to the new evaluation dataset;

in response to determining the performance of applying the filtered prediction model to the new evaluation dataset exceeds the performance of applying the complete prediction model to the new evaluation dataset:

designating as poisonous one or more data points, in both the training dataset and the full evaluation dataset, the one or more data points having a same provenance signature as:

the group of untrusted data points excluded from the second training dataset; and

the data points removed from the full evaluation dataset based on sharing the provenance signature with the one or more data points of the second training dataset; and

removing the one or more poisonous data points from the training dataset and the full evaluation dataset; and

recombining the training dataset and full evaluation dataset after removing the one or more poisonous data points therefrom so as to create a final filtered training set; and

training a final prediction model using the final filtered training set.