Smart de-identification using date jittering
System and method to produce an anonymized cohort having less than a predetermined risk of re-identification. The method includes receiving a data query of requested traits for the anonymized cohort, querying a data source to find records that possess at least some of the traits, forming a dataset from at least some of the records, and grouping the dataset in time into a first boundary group, a second boundary group, and one or more non-boundary groups temporally between the first boundary group and second boundary group. For each non-boundary group, calculating maximum time limits the non-boundary group can be time-shifted without overlapping an adjacent group, calculating a group jitter amount, capping the group jitter amount by the maximum time limits and by respective predetermined jitter limits, and jittering said non-boundary group by the capped group jitter amount to produce an anonymized dataset. Return the anonymized dataset.
1. A method of mitigating an inherent risk of re-identification from lengths of intervals between pairs of dates associated with a plurality of members in a dataset while preserving ordering of the dates, the method comprising steps of:
grouping, by a processor, the dates into a plurality of groups, the dates in each group being within a first predetermined time interval from each other and the groups being within a second predetermined time interval from each other, wherein the groups with the earliest and the latest dates are boundary groups and all other groups are non-boundary groups;
date shifting, by a processor, the dates in each of the plurality of groups together;
for each non-boundary group:
calculating, by the processor, forward and backward jitter limits of the forward and backward gaps adjacent to the group and forward and backward jitter shift parameters, respectively;
calculating, by the processor, using an earliest date in the group a group jitter amount; and
jittering, by the processor, said non-boundary group by the group jitter.
2. The method of claim 1 , further comprising capping the group jitter by respective predetermined configurable forward and backward jitter limits.
3. The method of claim 1 , wherein date shifting of the plurality of groups is performed together to maximize an associated analytic value.
4. The method of claim 2 , wherein the predetermined forward and backward jitter limits are dependent upon one or more characteristics of the data and are dependent intended uses of the data.
5. The method of claim 2 , wherein the group jitter amount is within a range established by the forward and backward jitter limits.
6. The method of claim 2 , wherein the predetermined forward and backward jitter limits are different for a first non-boundary group and a second non-boundary group.
7. The method of claim 1 , wherein the group jitter amount is calculated using a deterministic function.
8. The method of claim 2 , wherein the group jitter amount is calculated using a hash function.
9. The method of claim 8 , wherein the hash function output is scaled to the capped group jitter.
10. The method of claim 8 , wherein the hash function hashes a date value from each respective non-boundary group by use of a secret key.
11. The method of claim 1 , further comprising a step of calculating forward and backward minimum and maximum jitter limits as a function of a desired analytic value.
12. The method of claim 10 , wherein the secret key comprises a project sub-key specific to a project, and an entity sub-key specific to an entity whose data is being processed.
13. The method of claim 12 , wherein the project sub-key comprises a randomly-generated universally unique identifier (UUID).
14. The method of claim 1 , further comprising shifting the first boundary group, the second boundary group, and the one or more non-boundary groups by a predetermined amount.
15. The method of claim 1 , wherein the jittering is not performed on the boundary groups to guarantee incremental ordering, because only one adjacent gap can be calculated.
16. The method of claim 11 , wherein the minimum jitter limit represents allows the non-boundary group to be time-shifted without overlapping an adjacent group.
17. A system of mitigating an inherent risk of re-identification from lengths of intervals between pairs of dates associated with a plurality of members in a dataset while preserving ordering of the dates, the system comprising:
a processor configured to: group the dates into a plurality of groups, the dates in each group being within a first predetermined time interval from each other and the groups being within a second predetermined time interval from each other, wherein the groups with the earliest and the latest dates are boundary groups and all other groups are non-boundary groups;
date shift the dates in each of the plurality of groups together;
for each non-boundary group:
calculate forward and backward jitter limits of the forward and backward gaps adjacent to the group and forward and backward jitter shift parameters, respectively;
calculate using an earliest date in the group a group jitter amount;
jittering said non-boundary group by the group jitter.
18. The system of claim 17 , wherein the group jitter amount is calculated using a hash function.
19. The system of claim 18 , wherein the hash function hashes a date value from each respective non-boundary group by use of a secret key, wherein the secret key comprises a project sub-key specific to a project, and an entity sub-key specific to an entity whose data is being processed.
20. The system of claim 17 , further comprising shifting the first boundary group, the second boundary group, and the one or more non-boundary groups by a predetermined amount.