System and method for providing intelligent pre-staging of data in a compute environment
View Patent ↗A system, computer-readable medium and method for performing intelligent data pre-staging for a job submitted to a cluster environment. The method aspect comprises determining availability of compute resources including availability timeframes to process the submitted job, determining data requirements for processing the job and determining a co-allocation in time reservation.
1. A method comprising:
determining availability of compute resources in a compute environment including availability timeframes to process a workload submitted to the compute environment to yield a determined availability;
determining data requirements for processing the workload, the data requirements being associated with data to be retrieved from a storage medium for use by the workload, to yield determined data requirements;
based on the determined availability and the determined data requirements, establishing a data stage-in reservation associated with the data requirements for the workload to process that starts at a first start time, the data stage-in reservation only reserving resources associated with retrieving the data from the storage medium for use by the workload, wherein establishing the data stage-in reservation is based at least in part on (1) creating an availability range list from a calculation of existing resource guarantees and reservations and (2) converting the availability range list into a start range list;
performing data pre-staging according to the data stage-in reservation at the first start time; and
after performing the data pre-staging, processing the workload using the data and according to a compute reservation that starts at a second start time, which is later than the first start time.
2. The method of claim 1 , wherein establishing the data stage-in reservation further comprises:
(1) requesting resources for a first step in the workload based on at least one of the determined availability and the determined data requirements;
(2) requesting another resource for a next step in a workload process and returning to step (1) until all resources for all steps in the workload are requested;
(3) shifting start ranges in the start range list by an offset and performing an intersection operation on a combination start range;
(4) shifting the start ranges back by a negative of the offset, wherein resulting information provides when to start each potential data stage-in reservation; and
(5) establishing the data stage-in reservation based on the resulting information that processes when to start each potential data stage-in reservation.
3. The method of claim 2 , further comprising:
(1) presenting to a user a final list of selectable starting times for a data stage-in reservation; and
(2) upon receiving a user selection of a data stage-in reservation start time, establishing the data stage-in reservation.
4. The method of claim 1 , wherein compute resources are available prior to completion of data stage-in.
5. The method of claim 1 , wherein establishing the data stage-in reservation is based at least in part on an analysis of whether each step in the workload can be processed based on existing resource guarantees and reservations.
6. A system comprising:
a processor; and
a computer-readable medium storing instructions, which when executed by the processor, cause the processor to perform operations comprising:
determining data requirements for processing a workload, the data requirements being associated with data to be retrieved from a storage medium for use by the workload, to yield determined data requirements;
based on the determined availability and the determined data requirements, establishing a data stage-in reservation associated with the data requirements for the workload to process that starts at a first start time, the data stage-in reservation only reserving resources associated with retrieving the data from the storage medium for use by the workload, establishing the data stage-in reservation based at least in part on (1) creating an availability range list from a calculation of existing resource guarantees and reservations and (2) converting the availability range list into a start range list;
performing data pre-staging according to the data stage-in reservation at the first start time; and
after performing the data pre-staging, processing the workload using the data and according to a compute reservation that starts at a second start time, which is later than the first start time.
7. The system of claim 6 , wherein the instructions, which, when executed on the processor, further perform operations comprising:
establishing the data stage-in reservation by:
(1) requesting resources for a first step in the workload based on at least one of the determined availability and the determined data requirements;
(2) requesting another resource for a next step in a workload process and returning to step (1) until all resources for all steps in the workload are requested;
(3) shifting start ranges in the start range list by an offset and performing an intersection operation on a combination start range;
(4) shifting the start ranges back by a negative of the offset, wherein resulting information provides when to start each potential data stage-in reservation; and
(5) establishing the data stage-in reservation based on the resulting information that processes when to start each potential data stage-in reservation.
8. The system of claim 7 , wherein the instructions, which, when executed on the processor, further perform operations comprising:
presenting to a user a final list of selectable starting times for a data stage-in reservation and, upon receiving a user selection of a data stage-in reservation start time, establishing the data stage-in reservation.
9. The system of claim 6 , wherein compute resources are available prior to completion of data stage-in.
10. The system of claim 6 , wherein the instructions, which, when executed on the processor, further perform operations comprising:
establishing the data stage in reservation based on at least in part an analysis of whether each step in the workload can be processed based on existing resource guarantees and reservations.
11. A non-transitory computer-readable medium storing instructions for controlling a computing device, which, when executed on the computing device, perform operations comprising:
determining data requirements for processing a workload, the data requirements being associated with data to be retrieved from a storage medium for use by the workload, to yield determined data requirements;
based on the determined availability and the determined data requirements, establishing a data stage-in reservation associated with the data requirements for the workload to process that starts at a first start time, the data stage-in reservation only reserving resources associated with retrieving the data from the storage medium for use by the workload, wherein establishing the data stage-in reservation is based at least in part on (1) creating an availability range list from a calculation of existing resource guarantees and reservations and (2) converting the availability range list into a start range list;
performing data pre-staging according to the data stage-in reservation at the first start time; and
after performing the data pre-staging, processing the workload using the data and according to a compute reservation that starts at a second start time, which is later than the first start time.
12. The non-transitory computer-readable medium of claim 11 , wherein establishing the data stage-in reservation, further comprises:
(1) requesting resources for a first step in the workload based on at least one of the determined availability and the determined data requirements;
(2) requesting another resource for a next step in a workload process and returning to step (1) until all resources for all steps in the workload are requested;
(3) shifting start ranges in the start range list by an offset and performing an intersection operation on a combination start range;
(4) shifting the start ranges back by a negative of the offset, wherein resulting information provides when to start each potential data stage-in reservation; and
(5) establishing the data stage-in reservation based on the resulting information that processes when to start each potential data stage-in reservation.
13. The non-transitory computer-readable medium of claim 12 , the instructions further comprising:
(1) presenting to a user a final list of selectable starting times for the data stage-in reservation; and
(2) upon receiving a user selection of a data stage-in reservation start time, establishing the data stage-in reservation.
14. The non-transitory computer-readable medium of claim 11 , wherein compute resources are available prior to completion of data stage-in.
15. The non-transitory computer-readable medium of claim 11 , wherein establishing the data stage-in reservation is based at least in part on an analysis of whether each step in the workload can be processed based on existing resource guarantees and reservations.
16. The method of claim 1 , wherein establishing the data stage-in reservation is based on at least one of: a network speed, a delivered bandwidth, data cache usage, disk usage, storage capacity, virtual hosting center resources and a duration of network bandwidth resources.