Data staging management system
Batch job data staging combining synchronous/asynchronous staging. In pre-processing, a stage-in source file, and a target file for stage-out, in permanent storage, are identified using a batch script. From data amounts, time for stage-in/stage-out to/from temporary storage are estimated. Stage-in is based on the time, stage-out being asynchronous, and each asynchronous staging is classified short/long term depending on the time, each staging being recorded in a table. If a source file is modified, incremental staging is added to the table. With a staging list scheduling for batch jobs stage-in is performed, monitoring progress in the table, and resources may be allocated for the jobs nodes without waiting for stage-in to complete. The job generates results in the temporary storage, and using post-processing, stage-out transfers results to the target file in permanent storage.
1. A method of managing data staging, when executing, batch jobs in a computing system, comprising, for each batch job:
identifying at least one source file for data stage-in and at least one target file for data stage-out;
determining amounts of data to be transferred in data stage-in and data stage-out and estimating a time required to complete data transfer in data stage-in and data stage-out respectively;
setting, as a first data staging processing for the batch job, one of asynchronous and synchronous data stage-in based on the estimated time required for the data stage-in, and setting, as a second data staging processing for the batch job, asynchronous data stage-out, wherein the asynchronous data stage-in is carried out independently from resource allocation process without securing resources required for execution of the batch job and the synchronous data stage-in is carried out after securing the resource allocation for execution of the batch job, wherein the asynchronous data stage-out is carried out independently from resource allocation process;
performing data stage-in from the at least one source file and allocating resources for performing the batch job;
executing the batch job to generate results; and
performing data stage-out of the results to the target file.
2. The method according to claim 1 , further comprising:
classifying the first data staging processing as one of asynchronous short-term and asynchronous long-term data stage-in according to the estimated time required for the data stage-in; and
classifying the second data staging processing as one of asynchronous short-term and asynchronous long-term data stage-out according to the estimated time required for the data stage-out.
3. The method according to claim 2 , wherein the setting of the one of asynchronous and synchronous data stage-in is made by comparing the estimated time required for the data stage-in with a threshold value, and the allocating resources is performed once a remaining time to complete the data transfer is within the threshold value.
4. The method according to claim 3 , further comprising:
setting a maximum time for one of data stage-in and data stage-out;
calculating a multiplicity of respective data stage-in and data stage-out based on the maximum time; and
performing the respective data stage-in and data stage-out of a plurality of batch jobs simultaneously with a number of data staging processings equal to said multiplicity.
5. The method according to claim 4 , wherein the setting the maximum time includes forming a data staging management table recording each data staging processing and a status of each data staging processing.
6. The method according to claim 5 , further comprising:
while performing data stage-in, checking whether the source file has been modified since a start of data stage-in and if so, setting an incremental data stage-in as an additional data staging processing in the table.
7. The method according to claim 6 , wherein the data staging management table contains, for each data staging processing, initial information including:
an identifier of the batch job for which the data staging processing is set;
an indication of whether the data staging processing is the respective data stage-in and data stage-out;
a location of each source file and target file; and
an amount of data to be transferred.
8. The method according to claim 7 , wherein the initial information is obtained by analyzing a batch job script to perform said identifying and estimating.
9. The method according to claim 8 , wherein the data staging management table contains, for each data staging processing, further information including:
an indication of whether the data staging processing is an incremental data stage-in;
whether the data staging processing is one of has not started, is ongoing, and has completed;
a start time of data staging processing;
an end time of data staging processing; and
one of an amount of data so far transferred and an amount of data remaining to be transferred; and
an estimated time required for the data staging processing.
10. The method according to claim 9 , further comprising constructing, on a basis of the data staging management table, a data staging list for scheduling data stage-in or data stage-out of a plurality of batch jobs.
11. A data staging manager for managing data staging, in a computer system executing batch jobs, the data staging manager arranged to, for each batch job:
identify at least one source file for data stage-in, and at least one target file for data stage-out;
determine amounts of data to be transferred in data stage-in and data stage-out and estimate a time required for data transfer in data stage-in and data stage-out, respectively;
set, as a first data staging processing for the batch job, one of asynchronous and synchronous data stage-in based on the basis of the estimated time required for the data stage-in, and set, as a second data staging processing for the batch job, asynchronous data stage-out, wherein the asynchronous data stage-in is carried out independently from resource allocation process without securing resources required for execution of the batch job and the synchronous data stage-in is carried out after securing the resource allocation for execution of the batch job, wherein the asynchronous data stage-out is carried out independently from resource allocation process;
perform data stage-in from the at least one source file;
communicate with a job scheduler of the computer system to initiate allocation of resources to the batch job and execution of the batch job; and
perform data stage-out of results to the target file.
12. The data staging manager according to claim 11 , comprising a data staging management server, coupled to a pre-processor and a post-processor installed in the job scheduler of the computer system, the pre-processor arranged to analyze a batch job script received by the job scheduler.
13. The data staging manager according to claim 12 , wherein the data staging management server includes a data staging management table recording, for each data staging processing:
an identifier of the batch job for which the data staging processing is set;
an indication of whether the data staging processing is one of data stage-in and data stage-out;
a location of each source file and target file;
an amount of data to be transferred;
an indication of whether the data staging processing is an incremental data stage-in;
whether the data staging processing is one of has not started, is ongoing, and has completed;
a start time of data staging processing;
an end time of data staging processing;
one of an amount of data so far transferred and an amount of data remaining to be transferred; and
an estimated time required.
14. A non-transitory computer-readable recording media on which is recorded a computer program containing computer-readable instructions which, when executed by a server in a computer system, perform the method according to claim 1 .