Systems and methods for management of multi-tenancy data analytics platforms
A data analytics system configured to perform operations is disclosed. The operations can include creating, in response to instructions received from a user, a first pipeline. This pipeline can be configured to extract data from an append-only first data store, extract identifying characteristics from the extracted data, provide the identifying characteristics to an identity service, and receive a tenancy identifier from the identity service. The pipeline can further be configured to create a data object in a second data store using the extracted data; create a tenancy object in a metadata store, the tenancy object associated with the data object, the metadata store implementing a hierarchical data object ownership graph; and associate the tenancy object with a parent object in the hierarchical data object ownership graph. The data analytics system can then tear down the first pipeline.
1. A data analytics system, comprising:
at least one computer-readable medium containing instructions; and
at least one processor which executes the instructions to perform operations comprising:
creating a pipeline, in response to received instructions, configured to:
extract identifying characteristics from data in a first append-only data store;
provide the identifying characteristics to an identity service;
receive a tenancy identifier from the identity service;
create a data object in a second data store using the extracted identifying characteristics;
create a tenancy object associated with the data object and a parent object in a metadata store, wherein the tenancy object represents an ownership associated with the data object; and
upon associating the tenancy object with the data object, process a second set of data using the created pipeline.
2. The system of claim 1 , wherein:
the first pipeline is torn down following completion of the processing of the second set of data.
3. The system of claim 1 , wherein:
creating the pipeline is based on instructions provided by a user device.
4. The system of claim 1 , wherein:
the metadata store implements a hierarchical data object ownership graph.
5. The system of claim 1 , wherein:
the extracted identifying characteristics are received from a client device.
6. The system of claim 1 , wherein:
the pipeline is configured with location information or a function for generating an organization code using the extracted data.
7. The system of claim 1 , wherein the operations further comprise:
formatting information to convert data from one format to a second format.
8. The system of claim 1 , wherein:
the tenancy object and data object are associated based on a dependence of the locations of the data object or the tenancy identifier.
9. The system of claim 1 wherein the pipeline is further configured to:
allow a user permitted to access a parent object to access all data objects associated with the parent object.
10. The system of claim 1 wherein:
the operations further comprise disassociating the tenancy object from the first parent object and associating the tenancy object with a second parent object.
11. A data storage method, comprising:
creating a first pipeline, in response to received instructions from the data analytics system, configured to:
extract identifying characteristics from data in a first append-only data store;
provide the identifying characteristics to an identity service;
receive a tenancy identifier from the identity service;
create a data object in a second data store using the extracted identifying characteristics;
create a tenancy object associated with the data object and a parent object in a metadata store, wherein the tenancy object represents an ownership associated with the data object; and
upon associating the tenancy object with the data object, process a second set of data using the created pipeline.
12. The data storage method of claim 11 , wherein:
the first pipeline is torn down following completion of the processing of the second set of data.
13. The data storage method of claim 11 , wherein:
the steps of the method are instantiated by a user.
14. The data storage method of claim 11 , wherein:
the metadata store implements a hierarchical data object ownership graph.
15. The data storage method of claim 11 , wherein:
the extracted identifying characteristics are received from a client device.
16. The data storage method of claim 11 , wherein:
the pipeline is configured with location information or a function for generating an organization code using the extracted data.
17. The data storage method of claim 11 , wherein the operations further comprise:
formatting information to convert data from one format to a second format.
18. The data storage method of claim 11 , wherein:
the tenancy object and data object are associated based a dependence of the locations of the data object or the tenancy identifier.
19. The data storage method of claim 11 , wherein the pipeline is further configured to:
allow a user permitted to access a parent object to access all data objects associated with the parent object.
20. The data storage method of claim 11 , wherein:
the instructions further comprise disassociating the tenancy object from the first parent object and associating the tenancy object with a second parent object.