Self-contained system for de-identifying unstructured data in healthcare records

1. A method for de-identifying unstructured data within data sets, the method comprising:

initializing, using a computing device comprising a processor and memory, a blacklist dictionary having a structure that facilitates identification of personally identifiable information (PII) and protected health information (PHI) and that facilitates removal and replacement of PII and PHI with a case-type tag, and a whitelist dictionary comprising terms selected to remain in the data sets despite being included in the blacklist dictionary;

modifying, using the processor and a register module, the blacklist dictionary by removing terms included within the whitelist dictionary to create an adjusted blacklist dictionary;

augmenting, using the processor, the adjusted blacklist dictionary with a record-specific blacklist for each individual record within the data sets to create a record-specific adjusted blacklist dictionary;

scrubbing, using the processor and a de-identification engine, PII and PHI from each individual record utilizing the record-specific adjusted blacklist dictionary, the scrubbing comprising:

removing all elements within each individual record determined to be PII or PHI according to terms in the record-specific adjusted blacklist dictionary;

replacing removed elements with a case-type tag identifying a type of information being removed according to the record-specific adjusted blacklist dictionary; and

repeating, using the de-identification engine, the scrubbing of PII and PHI comprising removing steps and replacing steps for each individual record within the data sets.

2. The method of claim 1 , further comprising tokenizing and merging, using a merging module, each individual record in the data sets.

3. The method of claim 1 , wherein the blacklist dictionary comprises standard terms and standard number formats to be removed from records within the data sets.

4. The method of claim 3 , wherein the standard number formats comprise social security numbers, telephone numbers, URLs, zip codes, email addresses, IP addresses, dates, patient IDs, record numbers, and insurance IDs.

5. The method of claim 3 , wherein the standard terms comprise cities, counties, first names, last names, prefixes, and medical terms.

6. The method of claim 1 , wherein the record-specific adjusted blacklist dictionary comprises terms created from the PII or PHI present in specific individual records within the data sets.

7. The method of claim 1 , further comprising tuning terms in the whitelist dictionary and the record-specific adjusted blacklist dictionary to adjust a level of de-identification of records within the data sets.

8. The method of claim 7 , wherein the whitelist dictionary and the record-specific adjusted blacklist dictionary include a tunable list of names, birth dates, phone numbers, addresses and other forms of PII and PHI present in data store records or within the data sets.

9. The method of claim 7 , wherein augmenting comprises adjusting the adjusted blacklist dictionary to include known PII and PHI terms, for an individual associated with an individual record, according to the record-specific adjusted blacklist dictionary, that are designated to be removed.

10. A system for de-identifying unstructured data within data sets, comprising:

memory and a processor configured for accessing the data sets from data sources and parsing data within the data sets and identifying elements of unstructured data within the data sets and de-identifying unstructured data using a de-identification engine initializing a blacklist dictionary having a structure that facilitates identification of personally identifiable information (PII) and protected health information (PHI) and that facilitates removal and replacement of PII and PHI with a case-type tag, and a whitelist dictionary comprising terms selected to remain in the data sets despite being included in the blacklist dictionary, and comprising:

a register module configured to modify the blacklist dictionary by removing terms included within the whitelist dictionary to generate an adjusted blacklist dictionary, and augmenting the adjusted blacklist dictionary with a record-specific blacklist for each individual record within the data sets to create a record-specific adjusted blacklist dictionary;

a de-identification module configured to determine which unstructured elements are to be removed from each individual record of a data set by scrubbing PII and PHI from each individual record utilizing the record-specific adjusted blacklist dictionary, the scrubbing comprising:

removing all elements within each individual record determined to be PII or PHI according to terms in the record-specific adjusted blacklist dictionary; and

replacing removed elements with a case-type tag identifying a type of information being removed according to the record-specific adjusted blacklist dictionary;

wherein the de-identification module repeats the scrubbing PII and PHI for each individual record within the data sets; and

one or more user devices configured to receive input from one or more users by an input-output interface and communicate with the processor and the de-identification engine over a telecommunication network, providing functionality for the register module, de-identification module, and merging module that share a secured network connection.

11. The system of claim 10 , further comprising a merging module configured to tokenize and merge each individual record in the data sets by identifying a same token in both data sets and join together matched to individual records in healthcare data sets without exposure of PHI or PII.

12. The system of claim 10 , wherein the blacklist dictionary comprises standard terms and standard number formats to be removed from records within the data sets.

13. The system of claim 12 , wherein the standard number formats comprise social security numbers, telephone numbers, URLs, zip codes, email addresses, IP addresses, dates, patient IDs, record numbers, and insurance IDs.

14. The system of claim 12 , wherein the standard terms comprise cities, counties, first names, last names, prefixes, and medical terms.

15. The system of claim 10 , wherein the record-specific adjusted blacklist dictionary comprises terms created from the PII or PHI present in specific individual records within the data sets.

16. The system of claim 10 , further comprising tuning terms in the whitelist dictionary and the record-specific adjusted blacklist dictionary to adjust a level of de-identification of records within the data sets.

17. The system of claim 16 , wherein the whitelist dictionary and the record-specific adjusted blacklist dictionary include a tunable list of names, birth dates, phone numbers, addresses and other forms of PII and PHI present in data source records or within the data sets.

18. The system of claim 16 , wherein augmenting comprises adjusting the adjusted blacklist dictionary to include known PII and PHI terms for an individual associated with an individual record, according to the record-specific adjusted blacklist dictionary, that are designated to be removed.