Electronic medical record datasifter
A method is presented for generating a data set from a database. The method involves iterative data manipulation that stochastically identifies candidate entries from the cases (subjects, participants) and variables (data elements) and subsequently selects, nullifies, and imputes the information. This process heavily relies on statistical multivariate imputation to preserve the joint distributions of the complex structured data archive. At each step, the algorithm generates a complete dataset that in aggregate closely resembles the intrinsic characteristics of the original data set, however, on an individual level the rows of data are substantially altered. This procedure drastically reduces the risk for subject reidentification by stratification, as meta-data for all subjects is repeatedly and lossily encoded.
1. A computer-implemented method for generating a data set from a database, comprising:
receiving a set of records from a database, each record pertains to a particular entity and is comprised of a plurality of data fields;
identifying data fields in the set of records, wherein the data fields contain explicitly identifiable data for an entity;
obfuscating values in the data fields identified as containing explicitly identifiable data for the entity;
randomly selecting records in the set of records;
for each of the selected records, randomly selecting data fields in the selected records;
removing values in the selected data fields of the selected records in the set of records;
imputing values in data fields having missing values in the set of records using an imputation method; and
repeating the steps of removing values and imputing values for a number of iterations and thereby generate a new data set, where the number of iterations exceeds one.
2. The method of claim 1 further comprises obfuscating values in the data fields identified as containing explicitly identifiable data by removing the values from the identified data fields.
3. The method of claim 1 further comprises removing values in randomly selected data fields by randomly selecting data fields in the set of records using an alias method.
4. The method of claim 1 further comprises imputing values in data fields using multivariate imputation by chained equations.
5. The method of claim 1 further comprises, prior to the step of removing values in the data fields, identifying data fields in the set of records having unstructured data format and swapping values of at least one identified data field in a given pair of data records, where the given pair of data records is selected using a similarity measure between the data records in the set of records.
6. The method of claim 5 further comprises pairing a given data record in the set of records with each of the other data records in the set of records;
computing a similarity measure between values of data fields in each pair of records; and
identifying the given pair of records from amongst the pair of records, where the given pair of records has the highest similarity measure amongst the pair of records.
7. The method of claim 6 further comprises computing a similarity measures using a Bray-Curtis distance method.
8. The method of claim 5 further comprises
randomly selecting a given data field in the given data record;
pairing a given data record in the set of records with each of the other data records in the set of records;
computing a similarity measure between values of data fields in each pair of records;
forming a subset of data records from the set of records, where the records in the subset of data records have highest degree of similarity amongst the data records in the set of records;
randomly selecting a record from the subset of data records; and
swapping value of the given data field in the given data record with value in the corresponding data field on the randomly selected record from the subset of data records.
9. The method further comprises randomly selecting another data field in the given data record and, for the another data field in the given data record, repeating the steps of claim 8 .
10. The method further comprises retrieving another data record in the set of records and, for the another data record, repeating the steps of claim 8 .
11. A computer-implemented method for generating a data set from a database, comprising:
retrieving a set of records from a database, each record pertains to a particular person and is comprised of a plurality of data fields;
identifying data fields in the set of records, wherein the data fields that contain explicitly identifiable data for a person;
obfuscating values in the data fields identified as containing explicitly identifiable data for the person;
selecting at least one data field in a given data record in the set of records;
selecting a given pair of data records from the set of records, where the given pair of data records is selected using a similarity measure between the data records in the set of records;
swapping values of the at least one selected data field in the given pair of data records;
randomly selecting records in the set of records;
for each of the selected records, randomly selecting data fields in the selected records;
removing values in the selected data fields of the selected records in the set of records;
imputing values in data fields having missing values in the set of records using an imputation method; and
repeating the steps of removing values and imputing values for a number of iterations and thereby generate a new data set, where the number of iterations exceeds one.
12. The method of claim 11 further comprises obfuscating values in the data fields identified as containing explicitly identifiable data by removing the values from the identified data fields.
13. The method of claim 11 further comprises removing values in randomly selected data fields by randomly selecting data fields in the set of records using an alias method.
14. The method of claim 11 further comprises imputing values in data fields using multivariate imputation by chained equations.
15. The method of claim 11 wherein selecting a given pair of data records further comprises
pairing the given data record in the set of records with each of the other data records in the set of records;
computing a similarity measure between values of data fields in each pair of records; and
identifying the given pair of records from amongst the pair of records, where the given pair of records has the highest similarity measure amongst the pair of records.
16. The method of claim 15 further comprises computing a similarity measures using a Bray-Curtis distance method.
17. The method of claim 11 further comprises
randomly selecting a given data field in the given data record;
pairing a given data record in the set of records with each of the other data records in the set of records;
computing a similarity measure between values of data fields in each pair of records;
forming a subset of data records from the set of records, where the records in the subset of data records have highest degree of similarity amongst the data records in the set of records;
randomly selecting a record from the subset of data records; and
swapping value of the given data field in the given data record with value in the corresponding data field on the randomly selected record from the subset of data records.
18. The method further comprises randomly selecting another data field in the given data record and, for the another data field in the given data record, repeating the steps of claim 16 .
19. The method further comprises retrieving another data record in the set of records and, for the another data record, repeating the steps of claim 16 .