System and method for providing intelligent pre-staging of data in a compute environment
View Patent ↗A system, computer-readable medium and method for performing intelligent data pre-staging for a job submitted to a cluster environment. The method aspect comprises determining availability of compute resources including availability timeframes to process the submitted job, determining data requirements for processing the job and determining a co-allocation in time reservation.
1. A computer-implemented method of performing intelligent data pre-staging for a job submitted to a compute environment, the method comprising:
determining availability of compute resources in a cluster or grid environment including availability timeframes of the compute resources to process the submitted job;
determining data requirements for processing the job;
determining a co-allocation in time reservation by:
(1) requesting resources for a first step in the job process based on the determined availability and/or the determined data requirements;
(2) calculating existing resource guarantees and reservations already in place to create an availability range list;
(3) converting the availability range list into a start range list;
(4) requesting another resource for a next step in the job process and returning to step (2) until all resources for all steps in the job process are requested;
(5) shifting start ranges in the start range list by an offset and performing an intersection operation on a combination start range;
(6) shifting the start ranges back by a negative of the offset, wherein resulting information provides when to start each reservation;
(7) presenting to a user a final list of selectable starting times for a reservation; and
(8) upon receiving a user selection of a reservation start time, establishing a co-allocation in time reservation for resources for an appropriate start time; and
performing data pre-staging based on the co-allocation in time reservation and prior to processing the job in the compute environment.
2. The computer-implemented method of claim 1 , wherein the data requirements relate to a quantity of data and a speed of migration of the data to the compute resources.
3. The computer-implemented method of claim 1 , wherein the data requirement for processing the job are at least one of: network information, network speed, faults, statistical fluctuation, delivered bandwidth by the network and size.
4. The computer-implemented method of claim 1 , wherein the compute resources must be available prior to the completion of data pre-staging.
5. The computer-implemented method of claim 1 , wherein the range list indicates all the availability time frames.
6. A system for performing intelligent data pre-staging for a job submitted to a compute environment, the system comprising:
a processor;
a module configured to control the processor to determine availability of compute resources in a cluster or grid environment including availability timeframes to process the submitted job;
a module configured to control the processor to determine data requirements for processing the job;
a module configured to determine a co-allocation in time reservation, the module determining the co-allocation in time reservation by:
(1) requesting resources for a first step in the job process based on the determined availability and/or the determined data requirements;
(2) calculating existing resource guarantees and reservations already in place to create an availability range list;
(3) converting the availability range list into a start range list;
(4) requesting another resource for a next step in the job process and returning to step (2) until all resources for all steps in the job process are requested;
(5) shifting start ranges in the start range list by an offset and performing an intersection operation on a combination start range;
(6) shifting the start ranges back by a negative of the offset, wherein resulting information provides when to start each reservation;
(7) presenting to a user a final list of selectable starting times for a reservation; and
(8) upon receiving a user selection of a reservation start time, establishing a co-allocation in time reservation for resources for an appropriate start time; and
a module configured to control the processor to perform data pre-staging based on the co-allocation in time reservation and prior to processing the job in the compute environment.
7. The system of claim 6 , wherein the data requirements relate to a quantity of data and a speed of migration of the data to the compute resources.
8. The system of claim 6 , wherein the data requirement for processing the job are at least one of: network information, network speed, faults, statistical fluctuation, delivered bandwidth by the network and size.
9. The system of claim 6 , wherein the compute resources must be available prior to the completion of data pre-staging.
10. The system of claim 6 , wherein the range list indicates all the availability time frames.
11. A computer-readable medium containing instructions for controlling a computing device to perform intelligent data pre-staging for a job submitted to a compute environment, the instructions comprising:
determining availability of compute resources in a cluster or grid environment including availability timeframes of the compute resources to process the submitted job;
determining data requirements for processing the job;
determining a co-allocation in time reservation by:
(1) requesting resources for a first step in the job process based on the determined availability and/or the determined data requirements;
(2) calculating existing resource guarantees and reservations already in place to create an availability range list;
(3) converting the availability range list into a start range list;
(4) requesting another resource for a next step in the job process and returning to step (2) until all resources for all steps in the job process are requested;
(5) shifting start ranges in the start range list by an offset and performing an intersection operation on a combination start range;
(6) shifting the start ranges back by a negative of the offset, wherein resulting information provides when to start each reservation;
(7) presenting to a user a final list of selectable starting times for a reservation; and
(8) upon receiving a user selection of a reservation start time, establishing a co-allocation in time reservation for resources for an appropriate start time; and
performing data pre-staging based on the co-allocation in time reservation and prior to processing the job in the compute environment.
12. The computer-readable medium of claim 11 , wherein the data requirements relate to a quantity of data and a speed of migration of the data to the compute resources.
13. The computer-readable medium of claim 11 , wherein the data requirement for processing the job are at least one of: network information, network speed, faults, statistical fluctuation, delivered bandwidth by the network and size.
14. The computer-readable medium of claim 11 , wherein the compute resources must be available prior to the completion of data pre-staging.
15. The computer-readable medium of claim 11 , wherein the range list indicates all the availability time frames.