Privacy preserving federated query engine
A federated query engine system and method for multiple datasets is enhanced with privacy preserving features. It may, for example, limit the movement of data from one or more of the datasets being accessed. It may use cryptographic long-term keys, enabling fuzzy table joins that do not require a comparison of the plaintext column values. The query plan may leverage the particular infrastructure of the storage system that houses each of the datasets. The query engine receives a standard SQL query, translates the query into a logical plan for performing the query across the multiple datasets, converts the logical plan into physical plans that are specific to the implementational architecture of the multiple datasets, and sends these physical plans to SQL workers located near the data warehouses housing each dataset.
1 . A federated query method, comprising the steps of:
receiving a query statement, a schema for each of a plurality of datasets, and at least one privacy policy at a query engine, wherein each of the plurality of datasets comprises a plurality of rows and a plurality of columns;
parsing the query statement into a structured form;
based on the privacy policy, performing a structural transformation of the structured form of the query statement to comply with a set of privacy requirements in the privacy policy to produce a logical query plan, wherein the privacy policy comprises privacy limitations that are different between different datasets in the plurality of datasets, and wherein the structural transformation comprises applying long-term keys utilizing bloom-filter based cryptography that enables fuzzy matching in order to enable privacy-preserving table joins without requiring comparison of plaintext column values, and wherein the structural transformation comprises modifying the logical query plan to calculate counts of distinct rows contributing to aggregation groups, and the rows not meeting privacy thresholds are excluded;
based on the schema for each of the plurality of datasets and the logical query plan, generating at least one physical query plan for a query at each of the plurality of datasets; and
at each of a plurality of worker nodes, each of which corresponds to one of a plurality of data warehouses each housing one of the plurality of datasets, transforming the physical query plan into a query dialect appropriate to the dataset at the data warehouse that each of the plurality of worker nodes corresponds to in order to produce a translated query.
2 . The federated query method of claim 1 , wherein the step of parsing the query statement into a structured form comprises the step of parsing the query statement into a tree structure.
3 . The federated query method of claim 2 , wherein the step of parsing the query statement into a tree structure comprises the step of parsing the query statement into an abstract syntax tree.
4 . The federated query method of claim 1 , further comprising the step of receiving a location for each of the plurality of datasets.
5 . The federated query method of claim 1 , wherein the at least one physical plan comprises a plurality of physical plans.
6 . The federated query method of claim 5 , wherein each of the plurality of physical plans is applied to one of the plurality of datasets.
7 . The federated query method of claim 5 , further comprising the step of choosing a best physical plan from the plurality of physical plans.
8 . The federated query method of claim 1 , further comprising the step of running each of the translated queries against one of the plurality of datasets.
9 . The federated query method of claim 8 , further comprising the step of fetching the results of running each of the translated queries against each of the plurality of datasets back to the query engine.
10 . The federated query method of claim 9 , wherein the query engine comprises a coordinator, and further comprising the step of fetching the results of running each of the translated queries against each of the plurality of datasets back to the coordinator.
11 . The federated query method of claim 10 , further comprising the step of aggregating the results of running each of the translated queries against each of the plurality of datasets at the coordinator.
12 . The federated query method of claim 11 , further comprising the step applying a set of additional privacy constraints to the aggregated results.
13 . The federated query method of claim 1 , wherein the privacy policy comprises a query threshold value.
14 . The federated query method of claim 1 , wherein the privacy policy comprises a restriction on data movement of a portion of the data in at least one of the plurality of datasets.
15 . The federated query method of claim 14 , wherein the portion of the data in at least one of the plurality of datasets is a column in at least one of the plurality of datasets.
16 . The federated query method of claim 1 , wherein the privacy policy comprises application of data warehouse native privacy-enhancing features between joins of at least two of the plurality of datasets.
17 . The federated query method of claim 1 , wherein the step of performing a structural transformation of the structured form of the query statement to comply with a set of privacy requirements in the privacy policy to produce a logical query plan comprises the step of suppressing rows in the plurality of datasets below a threshold value.
18 . The federated query method of claim 17 , wherein the threshold value varies between at least two of the datasets in the plurality of datasets.