System and method for data migration
A data integration hub system and method is provided for migrating data from one or more source systems to a selection of target systems. Data migration is provided by a library of executable modules that are configured to transform input data to a form appropriate for a target data repository. The input data is made agnostic to the nature of the source of the data through a source bridge that is configured to access data from a variety of sources and convert the accessed data to a format acceptable to the executable modules. Embodiments also provide a mechanism for customizing the data integration hub by providing multiple development layers and restricting access to those layers.
1. A method for migrating data from a data source to a data target, the method comprising:
performing a lookup of access characteristics of the data source in response to a received request for data from a process performing the migration by executing one or more operations associated with executables that are stored in an executables library;
performing a request for the data from the data source wherein the requested data conforms to the access characteristics of the data source, wherein a set of source definitions is implemented to determine access to the source, wherein the data source includes different access formats for different data;
determining the data source and access mode upon receipt of the request for data by correlating the data request with entries in a source definition table, wherein the entries include source server address, identifier of the requested information, identifier of the information at the source server, and other access-related information associated with the source server;
processing data received from the data source in response to the request for data wherein said processing formats the data to an acceptable format required by the process performing the migration;
storing the processed data in a memory accessible to process performing the migration; and
performing data migration operations on the stored data that include format changes, combining with other data, permutations of the data, excluding data, and invoking applications to be used on the data.
2. The method for migrating data of claim 1 , wherein the access characteristics comprises one or more of source address, source field identifiers, and source protocol.
3. The method for migrating data of claim 2 , wherein the access characteristics are stored in a table having a plurality of entries each of which are associated with an associated data source.
4. The method for migrating data of claim 1 further comprising:
executing the process performing the data migration, wherein
the process comprises a task comprising an executable linked to the data source and a data target, an executable library accessible to the task stores the executable.
5. The method for migrating data of claim 4 further comprising:
accessing, by the executable of the task, the processed data;
performing additional processing on the data to place the data in a form associated with the data target;
providing the formatted data to the data target.
6. A method for migrating data from a data source to a data target, the method comprising:
defining a first migration task, wherein said defining comprises selecting an executable component and selecting one or more data target identifiers linked to one or more data source identifiers;
executing the first migration task is performed for each data target identifier and each data source identifier linked to the data target identifier, at a task executor module, wherein said executing comprises:
generating a data request for the selected data source; and
providing, by the task executor module, the data request to a source bridge;
requesting, by the source bridge, the data from the data source in a protocol and format required by the data source, wherein a set of source definitions is implemented to determine access to the source, wherein the data source includes different access formats for different data;
determining the data source and access mode upon receipt of the request for data by correlating the data request with entries in a source definition table, wherein the entries include source server address, identifier of the requested information, identifier of the information at the source server, and other access-related information associated with the source server;
providing, by the source bridge, the data to the task executor module in a format required by the executable component;
storing data in a memory accessible to the process performing data migration; and
performing data migration operations on the stored data that include format changes, combining with other data, permutations of the data, excluding data, and invoking applications to be used on the data.
7. The method for migrating data of claim 6 further comprising:
determining, by the source bridge, the format required by the data source by comparing information provided by the task and a table accessible to the source bridge that includes information associated with one or more data source servers and data fields.
8. The method for migrating data of claim 6 , wherein a first access level provides access to the one or more executable components, data source identifier, and data target identifier.
9. The method for migrating data of claim 8 further comprising providing a first user access to the first access level.
10. The method for migrating data of claim 8 , wherein
a second access level provides access to a second set of one or more executable components, data source identifiers, and data target identifiers,
an executable component of the second set of one or more executable components is not comprised in the first access level.
11. The method for migrating data of claim 10 further comprising providing a second user access to the second access level and not the first access level during modifying of a configuration of the second access level.