Creation of change-based data integration jobs
A computer software implemented method for transforming a first extract transform load (ETL) job having at least some unload transform load (UTL) portions. The method includes the following steps: (i) decomposing the first ETL job into an intermediate set of one or more jobs; and (ii) for each job of the intermediate set, transforming the job into a transactionally equivalent job to yield a final set of one or more jobs. The decomposing is performed so that each job of the intermediate jobs set is a Simple UTL job. The transforming is performed so that each job of the final set includes no UTL portions.
1. A method for transforming a first extract transform load (ETL) job having at least one unload transform load (UTL) portion, the method comprising:
decomposing the first ETL job into an intermediate set of one or more Simple UTL jobs;
for each Simple UTL job of the intermediate set, transforming the job into a transformationally equivalent job to yield a final set of one or more jobs that are a set of change-based transform load (CTL) job(s);
wherein:
the transforming includes the following (not necessarily in the following order):
replacing an input stage with a change capture stage programmed to receive change data from a database and to generate an output table including an indicator column that includes an indication of an insert type action or a delete type action,
delivering at least one update type action on a source database as a delete type action followed by an insert type action, and
performing at least one of the following: inserting a row in a target database table or deleting a row in a target database, and
running the set of CTL job(s) on a source database to yield a target database;
wherein:
each Simple UTL job of the intermediate set jobs meets the following criteria:
a single input action retrieves the contents of a single table, or view on a single table, in a database,
contains no actions that operate on the entirety of data for a job,
contains no actions that rely on ordering of data, and
at least one output stage operates on a database.
2. A computer program product method for transforming a first extract transform load (ETL) job having at least one unload transform load (UTL) portion, the computer program product comprising software stored on a software storage device in a non-transitory manner, the software comprising:
first program instructions programmed to decompose the first ETL job into an intermediate set of one or more Simple UTL jobs;
second program instructions programmed to, for each Simple UTL job of the intermediate set:
transform the job into a transformationally equivalent job to yield a final set of one or more jobs that are a set of change-based transform load (CTL) job(s);
replace an input stage with a change capture stage programmed to receive change data from a database and to generate an output table including an indicator column that includes an indication of an insert type action or a delete type action;
deliver at least one update type action on a source database as a delete type action followed by an insert type action; and
perform at least one of the following: inserting a row in a target database table or deleting a row in a target database;
third program instructions programmed to run the set of CTL job(s) on a source database to yield a target database;
wherein:
each Simple UTL job of the intermediate set jobs meets the following criteria:
a single input action retrieves the contents of a single table, or view on a single table, in a database,
contains no actions that operate on the entirety of data for a job,
contains no actions that rely on ordering of data, and
at least one output stage operates on a database; and
the software is stored on a software storage device in a manner less transitory than a signal in transit.
3. A computer system for transforming a first extract transform load (ETL) job having at least one unload transform load (UTL) portion, the computer system comprising:
a processor(s) set; and
a software storage device;
wherein:
the processor set is structured, located, connected or programmed to run software stored on the software storage device; and
the software comprises:
first program instructions programmed to decompose the first ETL job into an intermediate set of one or more Simple UTL jobs,
second program instructions programmed to, for each Simple UTL job of the intermediate set:
transform the job into a transformationally equivalent job to yield a final set of one or more jobs that a set of change-based transform load (CTL) job(s),
replace an input stage with a change capture stage programmed to receive change data from a database and to generate an output table including an indicator column that includes an indication of an insert type action or a delete type action,
deliver at least one update type action on a source database as a delete type action followed by an insert type action, and
perform at least one of the following: inserting a row in a target database table or deleting a row in a target database, and
third program instructions programmed to run the set of CTL job(s) on a source database to yield a target database;
wherein:
each Simple UTL job of the intermediate set jobs meets the following criteria:
a single input action retrieves the contents of a single table, or view on a single table, in a database,
contains no actions that operate on the entirety of data for a job,
contains no actions that rely on ordering of data, and
at least one output stage operates on a database.