Determining journalist risk of a dataset using population equivalence class distribution estimation
A system, method and computer readable memory for determining journalist risk of a dataset using population equivalence class distribution estimation. The dataset may be a cross-sectional data set or a longitudinal dataset. The determine risk of identification can be determined and used in de-identification process of the dataset.
1. A method of determining a journalist risk associated with a longitudinal dataset having personal data, including quasi-identifiers, of individuals representing a sample of a population in a database, the method comprising, on a computing device:
determining, from cross-sectional data in the longitudinal dataset for each of the individuals represented in the sample, a similarity measure;
rounding the similarity measure to an integer value, wherein using a combinatorics approach, the integer value of the rounded similarity measure directly translates to a respective one of equivalence classes, each including a subset of the individuals having identical values for all the quasi-identifiers, each subset having a subset size; and
for each of the equivalence classes:
obtaining a sample equivalence class distribution by dividing the rounded similarity measure by the determined similarity measure, wherein the obtained sample equivalence class distribution is a population equivalence class distribution;
calculating a probability that the equivalence class of a first size comes from population of the equivalence class of a second size;
calculating an average prosecutor risk, without consideration of the population, by dividing a proportion of the individuals that belong to the equivalence class by the subset size of the equivalence class and summing the result;
determining the journalist risk by multiplying the proportion of the individuals that belong to the equivalence class by the calculated probability;
de-identifying the database based on the journalist risk to provide a de-identified database; and
displaying the journalistic risk on a display of the computing device.
2. The method of claim 1 , wherein the calculating the probability that the equivalence class of the first size y in the population will be sampled to an equivalence class of the second size x in the sample, comprises computing:
Pr
(
size
x
in
sample
|
size
y
in
population
)
=
(
y
x
)
(
N
-
y
n
-
x
)
/
(
N
n
)
where n is a sample size, N is a population size and
(
y
x
)
=
y
!
x
!
(
y
-
x
)
!
.
3. The method of claim 2 , wherein the probability that the equivalence class is of the subset size y in the population and is the equivalence class of the subset size x in the population is:
Pr (size x in sample|size y in population)× Pr ( y )
where Pr(y)=a number of the equivalence classes of the subset size y in population/a total number of the equivalence classes in the population.
4. The method of claim 3 , wherein the determining the journalist risk comprises computing:
Risk
journalist
=
∑
i
=
1
m
ax
size
∑
j
=
i
ma
x
size
Pr
(
size
j
in
pop
.
|
size
i
in
samp
.
)
×
Prop
(
i
)
/
j
where Prop(i)=number of individuals in the equivalence classes of the subset size i/n.
5. The method of claim 1 , wherein the determining the journalist risk comprises:
matching personal data of someone from the population in the sample; and
computing
Risk
journalist
=
J
/
∑
i
=
1
ma
x
size
∑
j
=
i
ma
x
size
Pr
(
size
j
in
pop
.
|
size
i
in
samp
.
)
×
EC_num
(
i
)
×
j
where J is the number of equivalence classes in the sample and EC_num(i) is the number of equivalence classes of the subset size i in the sample.
6. A system to determine a journalist risk associated with a longitudinal dataset having personal data, including quasi-identifiers, of individuals representing a sample of a population in a database, the system comprising a processor configured to:
determine, from cross-sectional data in the longitudinal dataset for each of the individuals represented in the sample, a similarity measure;
round the similarity measure to an integer value, wherein using a combinatorics approach, the integer value of the rounded similarity measure directly translates into a respective one of equivalence classes, each including a subset of the individuals having identical values for all the quasi-identifiers, each subset having a subset size; and
for each of the equivalence classes:
obtain a sample equivalence class distribution by dividing the rounded similarity measure by the determined similarity measure, wherein the obtained sample equivalence class distribution is a population equivalence class distribution;
calculate a probability that the equivalence class of a first size comes from population of the equivalence class of a second size;
calculate an average prosecutor risk, without consideration of the population, by dividing a proportion of the individuals that belong to the equivalence class by the subset size of the equivalence class and summing the result;
determine the journalist risk by multiplying the proportion of records that belong to the equivalence class by the calculated probability;
de-identify the database based on the journalist risk to provide a de-identified database; and
displaying the journalistic risk on a display associated with the processor.
7. The system of claim 6 , wherein the processor is further configured to:
calculate the probability that an equivalence class of the subset size y in the population is sampled to an equivalence class of the subset size x in the sample, using:
Pr
(
size
x
in
sample
|
size
y
in
population
)
=
(
y
x
)
(
N
-
y
n
-
x
)
/
(
N
n
)
where n is the sample size, N is the population size and
(
y
x
)
=
y
!
x
!
(
y
-
x
)
!
.
8. The system of claim 7 , wherein the probability that the equivalence class is of the subset size y in the population and is the equivalence class of the subset size x in the population is:
Pr (size x in sample|size y in population)× Pr ( y )
where Pr(y)=a number of the equivalence classes of the subset size y in population/a total number of the equivalence classes in the population.
9. The system of claim 8 , wherein the journalist risk is determined by:
Risk
journalist
=
∑
i
=
1
ma
x
size
∑
j
=
i
ma
x
size
Pr
(
size
j
in
pop
.
|
size
i
in
samp
.
)
×
Prop
(
i
)
/
j
where Prop(i)=number of individuals in the equivalence classes of the subset size i/n.
10. The system of claim 6 , wherein the journalist risk is based upon matching personal data of someone from the population in the sample, and the processor is further configured a formula of:
Risk
journalist
=
J
/
∑
i
=
1
ma
x
size
∑
j
=
i
ma
x
size
Pr
(
size
j
in
pop
.
|
size
i
in
samp
.
)
×
EC_num
(
i
)
×
j
where J is the number of equivalence classes in the sample and EC_num(i) is the number of equivalence classes of the subset size i in the sample.
11. A non-transitory computer readable medium encoded with a computer program to determine a journalist risk associated with a longitudinal dataset having personal data, including quasi-identifiers, of individuals representing a sample of a population in a database, wherein the computer program comprises instructions to cause a processor to:
determine, from cross-sectional data in the longitudinal dataset for each of the individuals represented in the sample, a similarity measure;
round the similarity measure to an integer value, wherein using a combinatorics approach, the integer value of the rounded similarity measure directly translates into a respective one of equivalence classes, each including a subset of the individuals having identical values for all the quasi-identifiers, each subset having a subset size; and
for each of the equivalence classes:
obtain a sample equivalence class distribution by dividing the rounded similarity measure by the determined similarity measure, wherein the obtained sample equivalence class distribution is a population equivalence class distribution;
calculate a probability that the equivalence class of a first size comes from population of the equivalence class of a second size;
calculate an average prosecutor risk, without consideration of the population, by dividing a proportion of the individuals that belong to the equivalence class by the subset size of the equivalence class and summing the result;
determine the journalist risk by multiplying the proportion of records that belong to the equivalence class by the calculated probability;
de-identify the database based on the journalist risk to provide a de-identified database; and
display the journalistic risk on a display associated with the processor.
12. The non-transitory computer readable medium of claim 11 , further comprising instructions to cause the processor to:
calculate the probability that an equivalence class of the subset size y in the population is sampled to an equivalence class of the subset size x in the sample, using:
Pr
(
size
x
in
sample
|
size
y
in
population
)
=
(
y
x
)
(
N
-
y
n
-
x
)
/
(
N
n
)
where n is the sample size, N is the population size and
(
y
x
)
=
y
!
x
!
(
y
-
x
)
!
.
13. The non-transitory computer readable medium of claim 12 , wherein the probability that the equivalence class is of the subset size y in the population and is the equivalence class of the subset size x in the population is:
Pr (size x in sample|size y in population)× Pr ( y )
where Pr(y)=a number of the equivalence classes of the subset size y in population/a total number of the equivalence classes in the population.
14. The non-transitory computer readable medium of claim 13 , further comprising instructions to cause the processor to determine the journalist risk as:
Risk
journalist
=
∑
i
=
1
m
ax
size
∑
j
=
i
ma
x
size
Pr
(
size
j
in
pop
.
|
size
i
in
samp
.
)
×
Prop
(
i
)
/
j
where Prop(i)=number of individuals in the equivalence classes of the subset size i/n.
15. The non-transitory computer readable medium of claim 11 , wherein the journalist risk is based upon matching personal data of someone from the population in the sample, and the processor is further configured a formula of:
Risk
journalist
=
J
/
∑
i
=
1
ma
x
size
∑
j
=
i
ma
x
size
Pr
(
size
j
in
pop
.
|
size
i
in
samp
.
)
×
EC_num
(
i
)
×
j
where J is the number of equivalence classes in the sample and EC_num(i) is the number of equivalence classes of the subset size i in the sample.
16. The non-transitory computer readable medium of claim 11 , further including instructions to cause the processor to:
determine the journalistic risk based on optimal lattice anonymization.
17. The method of claim 1 , wherein the determining the journalistic risk comprises:
determining the journalistic risk based on optimal lattice anonymization.
18. The method of claim 6 , wherein the processor is further configured to:
determine the journalistic risk based on optimal lattice anonymization.
19. The method of claim 1 , wherein the database contains records of millions of people, and wherein the de-identifying comprises:
de-identifying the database containing records of millions of people based on the journalist risk to provide the de-identified database.
20. The system of claim 6 , wherein the database contains records of millions of people, and wherein the processor is further configured to:
de-identify the database containing records of millions of people based on the journalist risk to provide the de-identified database.
21. The non-transitory computer readable medium of claim 11 , wherein the database contains records of millions of people, further comprising instructions to cause the processor to:
de-identify the database containing records of millions of people based on the journalist risk to provide the de-identified database.