Centralized data management
A data management platform for managing interconnected data and its derivatives is disclosed. For one example of the present disclosure, the data management platform receives data assets to a data management platform through an API gateway. The data assets are reformatted based upon a corresponding data model. A set of data management features are accessed through a corresponding API. The set of data management modules includes tagging, ownership, relationship, cataloging, discovery, lineage and provenance, and lifecycle. The management modules provide dynamic identification of interconnections between the data assets. Interconnections for the data assets are generated and the data assets and the interconnection data are stored based upon a format of the data.
1. A centralized data management system, comprising:
an application program interface (API) gateway to receive, from various data
sources, data assets having multiple formats, and to reformat the data assets based upon a corresponding data model for a data asset;
plural data management modules, each plural data management module being independently deployable, and each plural data management being implemented as a microservice to dynamically identify interconnections
between the data assets, the data management modules providing at least two or more of tagging, ownership, relationship, cataloging, discovery, lineage and provenance, and lifecycle;
wherein the API gateway receives a data request, the data request including a set of data parameters that specify data and a set of relationship parameters corresponding to the data request and wherein one or more data management features of a set of data management features is used to identify interconnected data based upon the set of data parameters and the set of relationship parameters, further comprising: a user access device to present the data and the interconnected data to a user; and
a database to store the reformatted data assets and the interconnection data.
2. The centralized data management system of claim 1 wherein the interconnection data is metadata associated with a relationship between data assets.
3. The centralized data management system of claim 2 , wherein the metadata includes a data asset link that associates the data assets.
4. The centralized data management system of claim 1 wherein the data assets and the interconnection data are stored in a containerized database.
5. The centralized data management system of claim 1 wherein the data parameters include a data domain specifier and a data type specifier.
6. The centralized data management system of claim 1 wherein the data and the interconnected data are displayable in a format based upon the set of data parameter and the set of relationship parameters.
7. A non-transitory, computer-readable storage medium having stored thereon instructions which when executed by a processor, cause the processor to perform operations comprising: receiving data assets to a data management platform through an application program interface (API) gateway, the data assets having multiple formats; reformatting the data assets based upon a corresponding data model; accessing a set of data management services through a corresponding API, wherein each data management service is associated with its own API to dynamically identify interconnections between the data assets, the set of data management services including at least two or more tagging, ownership, relationship, cataloging, discovery, lineage and provenance, and lifecycle; generating interconnection data for the data assets, wherein the data management services are implemented as microservices, each data management service being independently deployable; and storing the data assets and the interconnection data;
wherein the instructions cause the processor to perform the further operation comprising: receiving a data request to the API gateway, the data request including a set of data parameters that specify data; receiving a set of relationship parameters corresponding to the data request; accessing one or more data management features of the set of data management features based upon the set of data parameters and the set of relationship parameters; identifying interconnected data based upon the specified data parameters and specified relation parameters; and presenting the data and the interconnected data to a user.
8. The non-transitory, computer-readable storage medium of claim 7 , wherein the interconnection data is metadata associated with a relationship between data assets.
9. The non-transitory, computer-readable storage medium of claim 8 , wherein the metadata includes a data asset link that associates the data assets.
10. The non-transitory, computer-readable storage medium of claim 7 , wherein the data assets and the interconnection data are stored in a containerized database.
11. The non-transitory, computer-readable storage medium of claim 10 , wherein the containerized database is a distributed graph database.
12. The non-transitory, computer-readable storage medium of claim 10 , wherein the containerized database is a distributed columnar database.
13. The non-transitory, computer-readable storage medium of claim 10 , wherein the containerized database is a distributed document-oriented database.
14. The non-transitory, computer-readable storage medium of claim 7 , wherein the instructions cause the processor to perform the further operation comprising:
receiving a data request to the API gateway, the data request including a set of data parameters that specify data;
receiving a set of relationship parameters corresponding to the data request;
accessing one or more data management features of the set of data management features based upon the set of data parameters and the set of relationship parameters;
identifying interconnected data based upon the specified data parameters and specified relation parameters; and
presenting the data and the interconnected data to a user.
15. The non-transitory, computer-readable storage medium of claim 14 , wherein the data parameters include a data domain specifier and a data type specifier.
16. The non-transitory, computer-readable storage medium of claim 14 , wherein the data and the interconnected data are presented to a user in a format based upon the set of data parameter and the set of relationship parameters.