Enterprise grade metadata and data mapping management application
View Patent ↗An enterprise grade data mapping application enables collaboration and reuse of many data integration components by leveraging metadata, enterprise transformations, and improves the process by creating automation, versioning, traceability, and impact analysis via its web enabled portal. This solution bridges this gap by leveraging metadata, and making the process automated, disciplined, predictable, and gives insight to the data lineage to both IT and business stakeholders via its web enabled portal. Mapping Manager brings standardization, collaboration, versioning, traceability, regulatory compliance and auditability, impact analysis, management visibility, and programmatic control where there is none now.
1. A data mapping manager system, comprising:
a metadata repository configured to store table and column names for use in data integration projects;
a transformation rules store configured to store transformation rules to be applied to source data before loading the source data into target destinations, for use in data integration projects; and
one or more processors configured to execute computer program modules, the computer program modules comprising:
a data integration project information input module configured to receive information relating to a data integration project comprising data source information and data target information;
a scanning and metadata retrieval module configured to scan at least one of a data source and a data target to locate metadata and extract the located metadata to the metadata repository;
a transformation rules generation and assignment module configured to generate transformation rules from user input and add them to the transformation rules store, to modify transformation rules in the transformation rules store based on user input, and to assign transformation rules to the data integration project based on user selections from the transformation rules store;
a data mappings generator module configured to generate data mappings between source data and target data for the data integration project using metadata extracted from the data source and the data target and the transformation rules assigned to the data integration project;
a data mappings organization and display module configured to organize and display data mappings for multiple data integration projects;
an agnostic ETL job generator module configured to auto-generate an ETL job for any of at least two ETL software tools based on the generated data mappings; and
a versioning module configured to save point-in-time snapshots of the generated data mappings and to show element-by-element comparisons between the saved snapshots of the generated data mappings.
2. The system of claim 1 , wherein the versioning module is further configured to save point-in-time snapshots of the transformation rules in the transformation rules store and to show element-by-element comparisons between the saved snapshots of the transformation rules.
3. A data mapping manager system, comprising:
a metadata repository configured to store table and column names for use in data integration projects;
a transformation rules store configured to store transformation rules to be applied to source data before loading the source data into target destinations, for use in data integration projects; and
one or more processors configured to execute computer program modules, the computer program modules comprising:
a data integration project information input module configured to receive information relating to a data integration project comprising data source information and data target information;
a scanning and metadata retrieval module configured to scan at least one of a data source and a data target to locate metadata and extract the located metadata to the metadata repository;
a transformation rules generation and assignment module configured to generate transformation rules from user input and add them to the transformation rules store, to modify transformation rules in the transformation rules store based on user input, and to assign transformation rules to the data integration project based on user selections from the transformation rules store;
a data mappings generator module configured to generate data mappings between source data and target data for the data integration project using metadata extracted from the data source and the data target and the transformation rules assigned to the data integration project;
a data mappings organization and display module configured to organize and display data mappings for multiple data integration projects;
an agnostic ETL job generator module configured to auto-generate an ETL job for any of at least two ETL software tools based on the generated data mappings; and
a versioning module configured to save point-in-time snapshots of the generated data mappings and to show element-by-element comparisons between the saved snapshots of the generated data mappings, wherein the computer program modules further comprise a lineage analyzer module, wherein the lineage analyzer module is configured to generate graphical displays for selected column names stored in the metadata repository, showing a progression of column names for data associated with the selected column names due to associated data integration projects and transformation rules applied in each associated data integration project.
4. The system of claim 3 , wherein the graphical displays further show underlying metadata properties for each column name in the progression.
5. The system of claim 1 , wherein the computer program modules further comprise an ETL job reverse engineering module configured to import an ETL job from an ETL software tool and extract associated data mappings.
6. The system of claim 1 , wherein the computer program modules further comprise an audit trail module, wherein the audit trail module is configured to track and audit every action performed by a user on the system surrounding metadata and data mapping changes.
7. The system of claim 1 , wherein the data mappings organization and display module is further configured to organize and display data mappings by data integration project and subject area in a project and mapping browser.
8. The system of claim 1 , wherein the data mappings generator module is further configured to automatically update the data mappings for the data integration project in accordance with changes made to at least one of the data source, data target, and assigned transformation rules, and wherein the versioning module is further configured to save a point-in-time snapshot of the generated data mappings when the data mappings are automatically updated by the data mappings generator module.
9. The system of claim 1 , wherein the computer program modules further comprise a reporting module configured to mine the metadata repository and transformation rules store to generate mapping intelligence and metadata reports, including at least one of system analysis reports, mapping specification reports, mapping validation reports, data dictionary reports, and impact analysis reports.
10. The system of claim 1 , wherein the data mappings generator module is further configured to generate data mappings between source data and multiple target locations for the data integration project.
11. A computer-implemented data mapping management method, the method being implemented in a computer system that includes one or more processors executing computer program modules, the method comprising:
receiving information relating to a data integration project comprising data source information and data target information;
scanning at least one of a data source and a data target, locating metadata and extracting the located metadata to a metadata repository;
generating transformation rules from user input and adding them to a transformation rules store, modifying transformation rules in the transformation rules store based on user input, and assigning transformation rules to the data integration project based on user selections from the transformation rules store;
generating data mappings between source data and target data for the data integration project using metadata extracted from the data source and the data target and the transformation rules assigned to the data integration project;
organizing and displaying data mappings for multiple data integration projects;
generating an ETL job for any of at least two ETL software tools based on the generated data mappings; and
saving point-in-time snapshots of the generated data mappings and showing element-by-element comparisons between the saved snapshots of the generated data mappings.
12. The method of claim 11 , further comprising saving point-in-time snapshots of the transformation rules in the transformation rules store and showing element-by-element comparisons between the saved snapshots of the transformation rules.
13. A computer-implemented data mapping management method, the method being implemented in a computer system that includes one or more processors executing computer program modules, the method comprising:
receiving information relating to a data integration project comprising data source information and data target information;
scanning at least one of a data source and a data target, locating metadata and extracting the located metadata to a metadata repository;
generating transformation rules from user input and adding them to a transformation rules store, modifying transformation rules in the transformation rules store based on user input, and assigning transformation rules to the data integration project based on user selections from the transformation rules store;
generating data mappings between source data and target data for the data integration project using metadata extracted from the data source and the data target and the transformation rules assigned to the data integration project;
organizing and displaying data mappings for multiple data integration projects;
generating an ETL job for any of at least two ETL software tools based on the generated data mappings; and
saving point-in-time snapshots of the generated data mappings and showing element-by-element comparisons between the saved snapshots of the generated data mappings, further comprising generating graphical displays for selected column names stored in the metadata repository showing a progression of column names for data associated with the selected column names due to associated data integration projects and transformation rules applied in each associated data integration project.
14. The method of claim 13 , wherein the graphical displays further show underlying metadata properties for each column name in the progression.
15. The method of claim 11 , further comprising importing an ETL job from an ETL software tool and extracting associated data mappings.
16. The method of claim 11 , further comprising tracking and auditing every action performed by a user on the system.
17. The method of claim 11 , wherein organizing and displaying data mappings for multiple data integration projects further comprises organizing and displaying data mappings by data integration project and subject area in a project and mapping browser.
18. The method of claim 11 , further comprising updating the data mappings for the data integration project in accordance with changes made to at least one of the data source, data target, and assigned transformation rules, and saving a point-in-time snapshot of the generated data mappings after updating the data mappings.
19. The method of claim 11 , further comprising mining the metadata repository and transformation rules store to generate mapping intelligence and metadata reports, including at least one of system analysis reports, mapping specification reports, mapping validation reports, data dictionary reports, and impact analysis reports.
20. The method of claim 11 , further comprising generating data mappings between source data and multiple target locations for the data integration project.