Federated decentralized data sharing
Methods and systems for secure, federated, and decentralized ownership, storage, sharing and usage of big data are provided. According to one example, a first server maintains access to a plurality of federated data sources including at least one local database and at least one remote database. The local database is subject to a first data policy controlled by the first server and the remote database is subject to a second data policy controlled by a remote server. The first server receives a query from a user electronic device and verifies at least one permission attribute of the user electronic device relative to the data policies. After the federated data operation query has been approved, the first server generates results of the query including data fields from the local and remote databases in compliance with the first and second data policies.
1 . A method comprising the steps of:
at a first server comprising a processor, a memory, and a network interface device connected to a network,
maintaining access to a plurality of federated data sources comprising at least one local database and at least one remote database wherein the at least one local database is subject to a first data policy controlled by the first server and the at least one remote database is subject to a second data policy controlled by a remote server;
receiving a federated data operation query from a user electronic device;
verifying at least one permission attribute of the user electronic device relative to the first data policy and relative to the second data policy;
responsive to the verifying, approving the federated data operation query based on the at least one permission attribute of the user electronic device; and
after the federated data operation query has been approved, running the federated data operation query to generate results of the federated data operation query comprising data fields from the at least one local database and the at least one remote database in compliance with the first data policy and the second data policy.
2 . The method of claim 1 further comprising transforming data fields from the at least one local database in compliance with the first data policy and transforming data fields from the at least one remote database in compliance with the second data policy.
3 . The method of claim 1 wherein the federated data operation query comprises one of an action for removing data fields, an action for modifying data fields, an action for exporting data fields to an external application, an action for querying data fields, an action for generating derived data fields, and an action for training a machine learning model.
4 . The method of claim 1 wherein the at least one local database comprises a data repository that manages and stores one of structured and un-structured data.
5 . The method of claim 1 wherein the federated data operation query comprises at least one of a query operation based on domain specific language (DSL), a scripting operation, and a machine learning operation.
6 . The method of claim 1 wherein the generating comprises one of a join operation, a select operation, and a scripting operation.
7 . The method of claim 6 wherein the generating comprises an inner join operation and the method further includes the step of aggregating one or more segments of the at least one remote database by the remote server.
8 . The method of claim 1 wherein the results of the federated data operation query comprise a machine learning training model.
9 . The method of claim 1 wherein the at least one remote database comprises a data segment.
10 . The method of claim 8 further comprising: caching a data segment in an auxiliary data structure in the at least one local database; and providing a preview of the federated data operation query to the user electronic device using the auxiliary data structure.
11 . At least one non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to:
maintain access to a plurality of federated data sources comprising at least one local database and at least one remote database wherein the at least one local database is subject to a first data policy controlled by a first server and the at least one remote database is subject to a second data policy controlled by a remote server;
receive a federated data operation query from a user electronic device;
verify at least one permission attribute of the user electronic device relative to the first data policy and relative to the second data policy;
responsive to the verifying, approve the federated data operation query based on the at least one permission attribute of the user electronic device; and
after the federated data operation query has been approved, run the federated data operation query to generate results of the federated data operation query comprising data fields from the at least one local database and the at least one remote database in compliance with the first data policy and the second data policy, wherein the generating comprises a join operation.