Method and system for executing a data integration application using executable units that operate independently of each other
Methods and systems for executing a data integration application using executable units that operate independently of each other. Methods include receiving a specification for a data integration application comprising programmatic rules and specifying data-flow, creating executable units corresponding to the programmatic rules, creating data-flow channels corresponding to the specified data-flow, and executing the executable units such that data is passed between them in a non-blocking manner.
1. A method of executing a data integration application using executable units, wherein the executable units are able to operate independently of each other, the method performed by one or more processors comprising:
a. receiving a specification for a data integration application, wherein the received specification specifies:
a set of computer-readable data sources from which data are to be extracted;
a set of computer-writable data targets into which data are to be stored;
a set of programmatic rules comprising extraction rules, which operate to extract data from a data source and to produce the extracted data as output, transformation rules, which operate to receive data as input and to produce transformed data as output, and storage rules, which operate to receive data as input and to store the received data into a data target; and
a set of directional data-flow segments that represent the data-flow among the programmatic rules, wherein each data-flow segment represents the unidirectional flow of data from a source rule to a destination rule;
b. creating executable units, wherein each executable unit corresponds to one of the programmatic rules and is initialized with computer logic operable to implement the corresponding programmatic rule;
c. creating data-flow channels, wherein each data-flow channel corresponds to one of the specified data-flow segments such that the data-flow channel is operable to convey data from the source rule of the data-flow segment to the destination rule of the data-flow segment;
d. launching the executable units, such that the executable units operate independently of each other to process data and to convey said data through the data-flow channels according to the specified data-flow, and such that each of the executable units provides status information that is independent of the data-flow channels.