Distance preserving hash method
A computer-implemented method of preparing an anonymised dataset for use in data analytics. The method includes the steps of: (a) labelling elements of a dataset to be analysed according to a labelling scheme; (b) selecting one or more labelled elements of the dataset to be replaced with a distance preserving hash; and for each selected element: (c) partitioning a data plane including the selected element into a plurality of channels, each channel covering a different distance space of the data plane; (d) hashing, using a cryptographic hash, data associated with the channel of the data plane in which the selected element resides, to form the distance preserving hash; and (e) replacing the selected element with the distance preserving hash.
1. A computer-implemented method of preparing an anonymised dataset for use in data analytics, the method including the steps of:
(a) labelling elements of a dataset to be analysed according to a labelling scheme;
(b) selecting a subsample from the dataset and deriving therefrom an accuracy threshold indicative of the distance between elements of data within the subsample;
(c) deriving, from the anonymised dataset, an estimated accuracy of distance measurement between elements of the anonymised dataset and comparing this estimated accuracy to the accuracy threshold;
(d) selecting one or more labelled elements of the dataset to be replaced with a distance preserving hash; and for each selected element:
(e) partitioning a data plane including the selected element into a plurality of channels, each channel covering a different distance space of the data plane;
(f) hashing, using a cryptographic hash, data associated with the channel of the data plane in which the selected element resides, to form the distance preserving hash; and
(g) replacing the selected element with the distance preserving hash.
2. The computer-implemented method of claim 1 , further comprising a step of sending, to an analytics server, the anonymised dataset including the distance preserving hash(es) once all of the selected elements have been replaced.
3. The computer-implemented method of claim 1 , wherein if the estimated accuracy is less than the accuracy threshold, the value of ω is increased and steps (e)-(f) are repeated.
4. The computer-implemented method of claim 1 , wherein the data associated with the channel of the data plane within which the selected element resides is a generalised value based on the content of the respective channel.
5. The computer-implemented method of claim 1 , wherein the data associated with the channel of the data plane in which the selected element resides is a channel identifier or code corresponding to the channel.
6. The computer-implemented method of claim 1 , wherein the labelling scheme is derived from a machine-learning classifier which operates over the dataset.
7. A computer-implemented method of analysing a dataset anonymised using the method of claim 1 , the method including the steps of:
receiving an anonymised dataset from each of a plurality of data source computers;
for each distance preserving hash, comparing the distance preserving hash with a respective distance preserving hash of another anonymised dataset; and
estimating, based on the comparison, a distance between one or more element(s) having the same label in each of the anonymised datasets.
8. The computer-implemented method of claim 7 , wherein each dimension of each multi-dimensional hash is compared with a respective dimension of the multi-dimensional hash of another anonymised dataset.
9. The computer-implemented method of claim 8 , wherein each of the dimensions of the multi-dimensional distance hashes is given a different weighting when the distance is estimated.
10. The computer-implemented method of claim 9 , wherein a higher weighting is given to a first dimension which is indicative of a smaller distance space than a lower weighting given to a second dimension which is indicative of a larger distance space.
11. The computer-implemented method of claim 1 , wherein the distance preserving hash is multi-dimensional distance preserving has having ω dimensions, each dimension being formed of C channels, each channel having a size γ, and wherein the steps (e)-(f) are repeated for each dimension of the multi-dimensional distance preserving hash.
12. The computer-implemented method of claim 11 , wherein ω is selected based on any one or more of: an accuracy metric, a privacy metric, and a distance metric.
13. The computer-implemented method of claim 11 , wherein ω is selected based on a quantization resolution metric, which is indicative of a distance from a given distance preserving hash to the associated selected element.
14. A system, including:
one or more data source computers; and
at least one analytics server;
wherein each of the one or more data source computers are configured to prepare an anonymised dataset for the analytics server by:
(a) labelling elements of a dataset to be analysed according to a labelling scheme;
(b) selecting a subsample from the dataset and deriving therefrom an accuracy threshold indicative of the distance between elements of data within the subsample;
(c) deriving, from the anonymised dataset, an estimated accuracy of distance measurement between elements of the anonymised dataset and comparing this estimated accuracy to the accuracy threshold;
(d) selecting one or more labelled elements of the dataset to be replaced with a distance preserving hash; and for each selected element:
(e) partitioning a data plane including the selected element into a plurality of channels, each channel covering a different distance space of the data plane;
(f) hashing, using a cryptographic hash, data associated with the channel of the data plane in which the selected element resides, to form the distance preserving hash;
(g) replacing the selected element with the distance preserving hash; and
(h) sending, to the analytics server, the anonymised dataset including the distance preserving hash once the contents of each selected element has been replaced.
15. A data source computer for preparing an anonymised dataset for an analytics server, the data source computer including memory and a processor, wherein the memory includes instructions which cause the processor to:
(a) label elements of a dataset to be analysed according to a labelling scheme;
(b) selecting a subsample from the dataset and deriving therefrom an accuracy threshold indicative of the distance between elements of data within the subsample;
(c) deriving, from the anonymised dataset, an estimated accuracy of distance measurement between elements of the anonymised dataset and comparing this estimated accuracy to the accuracy threshold;
(d) select one or more labelled elements of the dataset to be replaced with a distance preserving hash; and for each selected element:
(e) partition a data plane including the selected element into a plurality of channels, each channel covering a different distance space of the data plane;
(f) hash, using a cryptographic hash, data associated with the channel of the data plane in which the selected element resides, to form the distance preserving hash; and
(g) replace the selected element with the distance preserving hash.