Migrating data integration processes through use of externalized metadata representations
Methods and systems are provided for migrating a data integration facility, such as an ETL job, from a source data integration platform to a target data integration platform. Certain embodiments involve externalizing a metadata representation of a source data integration job; parsing the metadata representation; importing the parsed metadata into a plurality of object representations of the source data integration job; generating an intermediate representation of the source data integration platform based on the plurality of object representations; and translating the intermediate representation to generate a target data integration job; wherein the target data integration job is adapted perform substantially the same functions as the source data integration job.
1 . A method, comprising:
externalizing a metadata representation of a source data integration job;
parsing the metadata representation;
importing the parsed metadata into a plurality of object representations of the source data integration job;
generating an intermediate representation of the source data integration platform based on the plurality of object representations; and
translating the intermediate representation to generate a target data integration job;
wherein the target data integration job is adapted perform substantially the same functions as the source data integration job.
2 . The method of claim 1 wherein the source data integration job has a source native format.
3 . The method of claim 1 wherein the target data integration job has a target native format.
4 . The method of claim 3 wherein the source native format is different than the target native format.
5 . The method of claim 1 wherein the object representations comprise class/object representations.
6 . The method of claim 1 wherein the object representations comprise atomic representations.
7 . The method of claim 1 wherein the intermediate representation is stored in memory.
8 . The method of claim 1 wherein the source data integration job comprises an ETL job.
9 . The method of claim 1 wherein the metadata representations is in a format selected from the group consisting of an XML format, a Text Export format, a script format, a COBOL format, a C language format, a C++ format, and a Teradata format.
10 . The method of claim 1 wherein the step of externalizing a metadata representation includes storing items to be translated in memory to facilitate the process.
11 . The method of claim 1 wherein the step of generating an intermediate representation includes producing a set of objects that represent a generic meta-model for a data integration job.
12 . The method of claim 11 wherein the generic meta-model comprises an atomic meta-model.
13 . The method of claim 11 wherein the intermediate representation comprises a hub adapted to facilitate bi-directional translations.
14 . The method of claim 1 wherein the step of generating a virtual representation creates a bi-directional translation facility.
15 . The method of claim 1 wherein the source data integration job comprises a source instruction set.
16 . The method of claim 1 wherein the source data integration job comprises a source data integration function.
17 . The method of claim 1 wherein the source data integration job comprises a source data integration facility.
18 . The method of claim 1 wherein the source data integration job is associated with a data integration platform of at least one of a banking institution, a financial services institution, a health care institution, a hospital, an educational institution, a governmental institution, a corporate environment, a non-profit institution, a law enforcement institution, a manufacturer, a professional services organization, and a research institution.
19 . A method, comprising:
extracting an instruction set in a first format from a source ETL application instruction set file;
converting the instruction set into a plurality of representations in an externalized format;
parsing the plurality of representations;
transforming the plurality of representations into a generic model;
translating the generic model into the second format; and
loading the output of the translation into a destination ETL application instruction set file.
20 . The method of claim 19 wherein the step of parsing the plurality of representations comprises parsing metadata associated with the plurality of representations.
21 . The method of claim 20 wherein the metadata is in an XML format and the parsing is performed using an XML parser.
22 . The method of claim 19 wherein the generic model comprises at least one of a generic format, an object format, and an atomic format.
23 . The method of claim 19 wherein the method further comprises the step of testing the regenerated translated model.
24 . The method of claim 23 wherein the step of testing further comprises determining the effectiveness of the method.
25 . The method of claim 23 wherein the instruction set comprises at least one of an extract, a transform, and a load instruction set.
26 . A system comprising a computer facility adapted to:
externalize a metadata representation of a source data integration job;
parse the metadata representation;
import the parsed metadata into a plurality of object representations of the source data integration job;
generate an intermediate representation of the source data integration platform based on the plurality of object representations; and
translate the intermediate representation to generate a target data integration job;
wherein the target data integration job is adapted perform substantially the same functions as the source data integration job.
27 . The system of claim 26 wherein the source data integration job has a source native format.
28 . The system of claim 26 wherein the target data integration job has a target native format.
29 . The system of claim 28 wherein the source native format is different than the target native format.
30 . The system of claim 26 wherein the object representations comprise class/object representations.
31 . The system of claim 26 wherein the object representations comprise atomic representations.
32 . The system of claim 26 wherein the intermediate representation is stored in memory.
33 . The system of claim 26 wherein the source data integration job comprises an ETL job.
34 . The system of claim 26 wherein the metadata representations is in a format selected from the group consisting of an XML format, a Text Export format, a script format, a COBOL format, a C language format, a C++ format, and a Teradata format.
35 . The system of claim 26 wherein the computer facility is adapted to store items to be translated in memory.
36 . The system of claim 26 wherein the computer facility is adapted to generate an intermediate representation including a set of objects that represent a generic meta-model for a data integration job.
37 . The system of claim 36 wherein the generic meta-model comprises an atomic meta-model.
38 . The system of claim 36 wherein the intermediate representation comprises a hub adapted to facilitate bi-directional translations.
39 . The system of claim 26 wherein the computer facility is adapted to create a bi- directional translation facility.
40 . The system of claim 26 wherein the source data integration job comprises a source instruction set.
41 . The system of claim 26 wherein the source data integration job comprises a source data integration function.
42 . The system of claim 26 wherein the source data integration job comprises a source data integration facility.
43 . The system of claim 26 wherein the source data integration job is associated with a data integration platform of at least one of a banking institution, a financial services institution, a health care institution, a hospital, an educational institution, a governmental institution, a corporate environment, a non-profit institution, a law enforcement institution, a manufacturer, a professional services organization, and a research institution.
44 . A system, comprising a computer facility adapted to:
extract an instruction set in a first format from a source ETL application instruction set file;
convert the instruction set into a plurality of representations in an externalized format;
parse the plurality of representations;
transform the plurality of representations into a generic model;
translate the generic model into a second format; and
load an output of the translation into a destination ETL application instruction set file.
45 . The system of claim 44 wherein the computer facility is adapted to parse metadata associated with the plurality of representations.
46 . The system of claim 45 wherein the metadata is in an XML format and the parsing is performed using an XML parser.
47 . The system of claim 44 wherein the generic model comprises at least one of a generic format, an object format, and an atomic format.
48 . The system of claim 44 wherein the computer facility is further adapted to test the regenerated translated model.
49 . The system of claim 48 wherein testing includes determining an effectiveness of the output.
50 . The system of claim 48 wherein the instruction set comprises at least one of an extract instruction set, a transform instruction set, and a load instruction set.