Data management platform using metadata repository
An analytical computing environment for large data sets comprises a software platform for data management. The platform provides various automation and self-service features to enable those users to rapidly provision and manage an agile analytics environment. The platform leverages a metadata repository, which tracks and manages all aspects of the data lifecycle. The repository maintains various types of platform metadata including, for example, status information (load dates, quality exceptions, access rights, etc.), definitions (business meaning, technical formats, etc.), lineage (data sources and processes creating a data set, etc.), and user data (user rights, access history, user comments, etc.). Within the platform, the metadata is integrated with all platform services, such as load processing, quality controls and system use. As the system is used, the metadata gets richer and more valuable, supporting additional automation and quality controls.
1. A data management platform, comprising:
a shared hardware and software infrastructure comprising a distributed file system cluster for data storage; and
a management server operative to receive and manage data from a set of data sources as a data analytics environment, at least one data source of the set of data sources having base metadata associated therewith, the data being stored in the distributed file system cluster, the management server comprising a set of data management platform services to manage the data throughout its lifecycle in the data management platform, the management server operative to generate and maintain platform metadata for use in managing the data across the set of data management platform services, the platform metadata being distinct from the base metadata and comprising information exposed and available to each data management platform service during the lifecycle;
the management server configured to provide in-stream data profiling and data quality exception handling on the information as it is ingested and stored in the distributed file system cluster, the data quality exception handling flagging and holding for analysis any data records having errors or inconsistencies as determined by the associated base metadata;
wherein the data management services include a data shopping component that includes a shopping cart, the data shopping component responsive to receipt of inputs representing data fields being placed in the shopping cart together with receipt of a checkout command to automatically create a view of data associated with the data fields to facilitate a further interactive query, the data fields identifying at least first and second data sources that have distinct source system formats, wherein the view of data integrates data from the first and second data sources without changing source system formats; and
wherein the data management services further include a data preparation component configured during the further interactive query and with respect to the view to receive and process a set of one or more additional commands, thereby creating in real-time a custom data set, the one or more additional commands being one of: a join, a filter, and a transform operation.
2. The data management platform as described in claim 1 wherein the information comprises status information that is one of: data load dates, and data access rights.
3. The data management platform as described in claim 1 wherein the information comprises business and technical format definitions.
4. The data management platform as described in claim 1 wherein the information comprises lineage information.
5. The data management platform as described in claim 1 wherein the information comprises user data that is one of: user rights, access history, and user comment data.
6. The data management platform as described in claim 1 wherein the data management platform services include a load processing service to receive transfer of data from a data source, and to perform a quality control on the transferred data.
7. The data management platform as described in claim 6 wherein the data management platform services include a source processing service to format and profile data received from the load processing service.
8. The data management platform as described in claim 7 wherein the data management platform services include a subject processing service to standardize and transform data received from the source processing service.
9. The data management platform as described in claim 1 wherein the data management platform services include an analytics processing service to format data to facilitate an analysis.
10. The data management platform as described in claim 1 wherein the data management platform services include an access service to enable receipt of a query with respect to the data.
11. The data management platform as described in claim 1 further including a graphical user interface (GUI) that exposes a shop-for-data function, wherein receipt of a selection of the shop-for-data function enables identification of data on which a data analysis is then performed and reported via the GUI.
12. The data management platform as described in claim 11 wherein the interface provides a browse and search function with respect to one of: the data, and the platform metadata.
13. The data management platform as described in claim 11 wherein the GUI provides a browse, search and select function to enable definition of a given subset of the data or platform metadata for analysis.
14. The data management platform as described in claim 13 wherein the interface is operative to create a view of selected data for interactive querying, the view providing a tracking of data lineage that includes data sources and calculations used to create the selected data, the data lineage being displayable upon receipt of a user query.
15. The data management platform as described in claim 13 wherein the interface includes an interactive tool to design a custom data flow.
16. The data management platform as described in claim 12 wherein information identified by the search and browse function is adapted to be tagged to facilitate a subsequent collaboration with respect to the tagged data.
17. The data management platform as described in claim 1 wherein the data management platform services include a data protection service.
18. The data management platform as described in claim 17 wherein the data protection service compresses and encrypts given data, wherein the given data is compressed at a block-based level.
19. The data management platform as described in claim 1 wherein the distributed file system cluster is Hadoop and data received is stored in at least one data lake.
20. The data management platform as described in claim 1 wherein first and second data analytics environments are instantiated by the management server, the first data analytics environment associated with a first enterprise, and the second data analytics environment associated with a second enterprise distinct from the first enterprise.