System and method for partitioning data based on authorization rules
A system and method for partitioning data based on authorization rules. The method includes initiating a plurality of data nodes, each data node including a dataset generated based on a shared data model, wherein each of the plurality of data nodes corresponds to at least one authorization rule, wherein each authorization rule authorizes at least one user identifier to access a respective data node of the plurality of data nodes; and configuring an extract, transform, and load (ETL) application service to extract data from a data source and to load data from the data source to the dataset stored on a first data node of the plurality of data nodes when the at least one first authorization rule authorizes a requesting user to access the first data node.
1. A method for partitioning data based on authorization rules, comprising:
initiating a plurality of data nodes, each data node including a dataset generated based on a shared data model, wherein each of the plurality of data nodes corresponds to at least one authorization rule, wherein each authorization rule authorizes at least one user identifier to access a respective data node of the plurality of data nodes; and
prior to receiving a user request for data, initially configuring an extract, transform, and load (ETL) application service to extract data from a data source and to load data from the data source to the dataset stored on a first data node of the plurality of data nodes when the at least one first authorization rule authorizes a requesting user to access the first data node.
2. The method of claim 1 , further comprising:
establishing a plurality of ETL pipelines, each ETL pipeline corresponding to one of the plurality of data nodes, wherein data filtered through each of the ETL pipelines is loaded into the dataset stored on the data node corresponding to the ETL pipeline.
3. The method of claim 2 , wherein new data to be stored on the plurality of data nodes is continuously filtered through the plurality of ETL pipelines.
4. The method of claim 2 , wherein each ETL pipeline has instructions for performing ETL operations, wherein the instructions of each ETL pipeline includes instructions for loading data into the corresponding data node.
5. The method of claim 1 , wherein the data source is a first data source of a plurality of data sources, wherein the shared data model includes a plurality of portions, wherein each of the plurality of portions is stored in a different data source of the plurality of data sources.
6. The method of claim 1 , wherein each authorization rule further authorizes the respective at least one user identifier to access at least one location of data.
7. The method of claim 6 , wherein each authorization rule defines a key, wherein the at least one location of data authorized by each authorization rule stores the key defined in the authorization rule.
8. The method of claim 1 , further comprising:
generating a plurality of user groups, wherein each of the plurality of user groups includes at least one user corresponding to the at least one user identifier of one of the at least one authorization rule.
9. A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process, the process comprising:
initiating a plurality of data nodes, each data node including a dataset generated based on a shared data model, wherein each of the plurality of data nodes corresponds to at least one authorization rule, wherein each authorization rule authorizes at least one user identifier to access a respective data node of the plurality of data nodes; and
prior to receiving a user request for data, initially configuring an extract, transform, and load (ETL) application service to extract data from a data source and to load data from the data source to the dataset stored on a first data node of the plurality of data nodes when the at least one first authorization rule authorizes a requesting user to access the first data node.
10. A system for partitioning data based on authorization rules, comprising:
a processing circuitry; and
a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:
initiate a plurality of data nodes, each data node including a dataset generated based on a shared data model, wherein each of the plurality of data nodes corresponds to at least one authorization rule, wherein each authorization rule authorizes at least one user identifier to access a respective data node of the plurality of data nodes; and
prior to receiving a user request for data, initially configure an extract, transform, and load (ETL) application service to extract data from a data source and to load data from the data source to the dataset stored on a first data node of the plurality of data nodes when the at least one first authorization rule authorizes a requesting user to access the first data node.
11. The system of claim 10 , wherein the system is further configured to:
establish a plurality of ETL pipelines, each ETL pipeline corresponding to one of the plurality of data nodes, wherein data filtered through each of the ETL pipelines is loaded into the dataset stored on the data node corresponding to the ETL pipeline.
12. The system of claim 11 , wherein new data to be stored on the plurality of data nodes is continuously filtered through the plurality of ETL pipelines.
13. The system of claim 11 , wherein each ETL pipeline has instructions for performing ETL operations, wherein the instructions of each ETL pipeline includes instructions for loading data into the corresponding data node.
14. The system of claim 10 , wherein the data source is a first data source of a plurality of data sources, wherein the shared data model includes a plurality of portions, wherein each of the plurality of portions is stored in a different data source of the plurality of data sources.
15. The system of claim 10 , wherein each authorization rule further authorizes the respective at least one user identifier to access at least one location of data.
16. The system of claim 15 , wherein each authorization rule defines a key, wherein the at least one location of data authorized by each authorization rule stores the key defined in the authorization rule.
17. The system of claim 10 , wherein the system is further configured to:
generate a plurality of user groups, wherein each of the plurality of user groups includes at least one user corresponding to the at least one user identifier of one of the at least one authorization rule.