Systems and methods for obscuring data from a data source
Systems and methods for obscuring data from a data source include devices and processes that may objectively measure the information loss for a dataset that is caused by applying a privacy policy, and may select and apply a policy to the dataset based on the measured information loss. The systems and methods may measure the information loss for a large dataset by taking a representative sample from the dataset and applying the policy to the sample in order to quantify the information loss. The quantified information loss can be iteratively used to change the policy in order to meet utility and/or privacy goals, and the system can subsequently apply the changed policy to the dataset.
1. A system for measuring an impact of a data privacy policy on a dataset, the system comprising:
a computer-readable data storage device containing program instructions; and
a processor, operably connected to the computer-readable data storage device, that executes the program instructions to perform operations comprising:
sampling a data source to obtain a sample of data that is statistically representative;
determining ridge statistics for the sample of data;
measuring a first entropy of the sample of data;
applying the data privacy policy to the sample of data;
measuring a second entropy of the sample of data with the policy applied;
calculating an information loss value based on the difference between the first entropy and the second entropy; and
presenting the information loss value.
2. The system of claim 1 , wherein the operations further comprise:
applying the data privacy policy to the data source if the information loss value is within predetermined limits.
3. A system for measuring information loss, the system comprising:
a computer-readable data storage device containing program instructions; and
a processor, operably connected to the computer-readable data storage device, that executes the program instructions to perform operations comprising:
sampling a data source to obtain a sample of data that is statistically representative;
determining ridge statistics for the sample of data;
receiving a data privacy policy for a dataset of the data source;
determining an estimate of information loss caused by the data privacy policy using the ridge statistics;
displaying the estimate of the information loss; and
applying the data privacy policy to the data source if the estimate of the information loss is within predetermined limits.