PLUG-IN ARCHITECTURE FOR USING EXTERNAL TOOLS WITH ETL PRODUCTS
According to some embodiments, a method and an apparatus of interfacing an external data processing tool (“DPT”) with an Extract, Transform and Load (“ETL”) product comprise defining a DPT to ETL schema and defining an ETL to DPT schema. The DPT may be called via a generic external transform using a plurality of DPT parameters. Data may be transmitted to the DPT, via a processor, based on the ETL to DPT schema and the DPT parameters.
1 . A method of interfacing an external data processing tool (“DPT”) with a Extract, Transform and Load (“ETL”) product, the method comprising:
defining an ETL to DPT schema;
defining a DPT to ETL schema;
calling the DPT via a generic external transform using a plurality of DPT parameters; and
transmitting data to the DPT, via a processor, based on the ETL to DPT schema and the DPT parameters.
2 . The method of claim 1 , further comprising:
receiving transformed data from the DPT, the transformed data based on a function of the DPT and the transmitted data to the DPT.
3 . The method of claim 1 , wherein the DPT parameters comprise:
a command line associated with the DPT;
a name of the ETL to DPT schema file;
a name of a metadata conversion script associated with the ETL to DPT schema file;
a mode;
a name of the DTP to ETL schema file; and
a name of a metadata conversion script associated with the DPT to ETL schema file.
4 . The method of claim 3 , wherein the parameters comprise hard-coded parameters.
5 . The method of claim 3 , wherein the command line associated with the DPT comprises:
dynamic parameters.
6 . The method of claim 3 , wherein the mode comprises a type of communication.
7 . The method of claim 6 , wherein the type of communication comprises (i) a file (ii) a filter, (iii) socket connection, or (iv) shared memory.
8 . A non-transitory computer-readable medium comprising instructions that when executed by a processor perform a method of interfacing an external data processing tool (“DPT”) with a Extract, Transform and Load (“ETL”) product, the method comprising:
defining an ETL to DPT schema;
defining a DPT to ETL schema;
calling the DPT via a generic external transform using a plurality of DPT parameters; and
transmitting data to the DPT, via a processor, based on the ETL to DPT schema and the DPT parameters.
9 . The medium of claim 8 , further comprising:
receiving transformed data from the DPT, the transformed data based on a function of the DPT and the transmitted data to the DPT.
10 . The medium of claim 8 , wherein the DPT parameters comprise:
a command line associated with the DPT;
a name of the ETL to DPT schema file;
a name of a metadata conversion script associated with the ETL to DPT schema file;
a mode;
a name of the DTP to ETL schema file; and
a name of a metadata conversion script associated with the DPT to ETL schema file.
11 . The medium of claim 10 , wherein the command line associated with the DPT comprises:
hard-coded parameters.
12 . The medium of claim 10 , wherein the command line associated with the DPT comprises:
dynamic parameters.
13 . The medium of claim 10 , wherein the mode comprises a type of communication.
14 . The medium of claim 13 , wherein the type of communication comprises (i) a file (ii) a filter, (iii) socket connection, or (iv) shared memory.
15 . An apparatus comprising:
a processor; and
a non-transitory computer-readable medium comprising instructions that when executed by a processor perform a method of interfacing an external data processing tool (“DPT”) with a Extract, Transform and Load (“ETL”) product, the method comprising:
defining an ETL to DPT schema;
defining a DPT to ETL schema;
calling the DPT via a generic external transform using a plurality of DPT parameters; and
transmitting data to the DPT, via a processor, based on the ETL to DPT schema and the DPT parameters.
16 . The apparatus of claim 15 , wherein the method further comprises:
receiving transformed data from the DPT, the transformed data based on a function of the DPT and the transmitted data to the DPT.
17 . The medium of claim 15 , wherein the DPT parameters comprise:
a command line associated the DPT;
a name of the ETL to DPT schema file;
a name of a metadata conversion script associated with the ETL to DPT schema file;
a mode;
a name of the DTP to ETL schema file; and
a name of a metadata conversion script associated with the DPT to ETL schema file.
18 . The apparatus of claim 17 , wherein the command line associated with the DPT comprises:
hard-coded parameters.
19 . The apparatus of claim 17 , wherein the command line associated with the DPT comprises:
dynamic parameters.
20 . The apparatus of claim 15 , wherein a number of input records, a number of output records and a total processing time will be displayed in response to calling the DPT via a generic external transform.