System and method for flexible data transfer
A method and system for flexibly transferring data from one or more data sources to one or more data destinations within an information network where each of the one or more data sources have data in a particular source format and each of the one or more data destinations have data in the same or another particular destination format using a parameter database that includes parameters to control the transfer of data, a scheduler that initiates the transfer of data, and a data loader in communications with the parameter database and scheduler that, upon initiation by the scheduler, extracts data from the one or more data sources, manipulates the extracted source data into one or more destination formats associated with the one or more data destinations, and inserts the data into one or more data destinations according to the parameters within the parameter database.
1. A method of transferring data from a source to a destination, the method comprising:
receiving, at a data loader, an indication of a data source;
receiving an indication of a data destination;
determining a first value indicative of a geographic distance between the data source and the data loader;
determining a second value indicative of a geographic distance between the data destination and the data loader;
determining a number of threads to be used by the data loader to transfer a source data object, wherein the number of threads is determined based, at least in part, on a geographic distance value selected from a group consisting of the first value and the second value; and
employing the number of threads to transfer the source data object from the data source to the data destination;
wherein the number of threads is determined to be one when the second value is greater than the first value; and
wherein the number of threads is determined to be one when the second value is greater than the first value.
2. A method as defined in claim 1 , wherein each of the number of threads employed to transfer the source data object is initialized simultaneously.
3. A method as defined in claim 1 , wherein the data source is a first data source and wherein the method further includes receiving an indication of a second data source.
4. A method as defined in claim 3 , wherein employing the number of threads to transfer the source data object comprises employing a first thread to transfer data from the first data source and employing a second thread to transfer data from the second data source.
5. A data loader server, comprising:
a processor having access to a computer readable storage medium storing processor executable program instructions for transferring data from a source to a destination, the program instructions including instructions for:
determining a first value indicative of a geographic distance between a data source and the data loader;
determining a second value indicative of a geographic distance between a data destination and the data loader;
determining a number of transfer threads based on a geographic distance value selected from a group consisting of the first value and the second value; and
employing the number of threads to transfer a data object from the data source to the data destination;
wherein the number of transfer threads is determined to be one when the second value is greater than the first value; and
wherein the number of transfer threads is determined to be greater than one when the first value is greater than the second value.
6. The data loader server of claim 5 , wherein each of the number of threads are initialized simultaneously.
7. The data loader server of claim 5 , wherein the data object is a first data object and wherein employing the number of threads includes:
employing a first transfer thread to transfer at least a portion of the first data object from the first data source; and
employing a second transfer thread to transfer at least a portion of a second data object from a second data source.
8. An article of manufacture storing machine readable instructions which, when executed, cause a machine to:
receive a first parameter identifying a data source;
receive a second parameter identifying a data destination;
determine a first value indicative of a geographic distance between a data source and a data loader;
determine a second value indicative of a geographic distance between a data destination and the data loader;
determine a number of data transfer threads corresponding to a source data structure based, at least, on the first value and the second value; and
employ the number of data transfer threads to transfer the source data structure from the data source to the data destination;
wherein the number of threads is one when the second value is greater than the first value; and
wherein the number of threads is greater than one when the first value is greater than the second value.
9. The article of manufacture of claim 8 , wherein each of the number of threads is initialized simultaneously.
10. The article of manufacture of claim 8 , wherein the data source is a first data source, the source data structure is a first source data structure, and the instructions include instructions to determine a second source value indicative of a distance between a second data source and the data loader.
11. The article of manufacture of claim 10 , wherein the instructions to employ include instructions to employ a first transfer thread to transfer at least a portion of the first source data structure and a second transfer thread to transfer at least a portion of a second source data structure from the second data source.