Optimizing job preemption using dynamic and static information
In an approach for optimizing job preemption using a scheduling mechanism that considers dynamic and static information, a processor, responsive to a pending job being selected to be run, determines that no computing hosts within a computing cluster have available resources capable of running the pending job and non-preemptive measures will not provide required resources for the pending job. A processor identifies candidate jobs for preemption and dynamic information and static information for each candidate job. A processor ranks the candidate jobs for preemption based on the dynamic information and the static information for each candidate job. A processor attempts to preempt a top N candidate jobs whose released resources in combination with the available resources of the computing cluster satisfy the required resources of the pending job. A processor, responsive to successfully preempting the top N candidate jobs, initiates the pending job.
1 . A computer-implemented method comprising:
responsive to a pending job being selected to be run, determining, by one or more processors, that no computing hosts within a computing cluster have available resources capable of running the pending job and non-preemptive measures will not provide required resources for the pending job, wherein determining that no computing hosts within the computing cluster have available resources capable of running the pending job and the non-preemptive measures will not provide the required resources for the pending job comprises:
reviewing current jobs running and available resources within the computing cluster at a point in time by taking a snapshot; and comparing the available resources within the computing cluster to the required resources to run the pending job;
identifying, by the one or more processors, candidate jobs for preemption and dynamic information and static information for each candidate job, wherein the static information is a configured priority value assigned to a respective candidate job and pulled from a job submission script, wherein the dynamic information is identified by polling the computing cluster for the dynamic information about each candidate job as that candidate job runs in the computing cluster, wherein the one or more processors maintain, in a job scheduler component:
(i) a first data structure that stores, for each running job within the computing cluster, a latest checkpoint time detected for that running job, the latest checkpoint time being recorded explicitly using a scheduler application programming interface (API) by which the running job marks a successful completion of a checkpoint, and
(ii) a second data structure that stores, for each running job within the computing cluster, a predicted total time for completion and a start time for that running job, wherein the dynamic information includes:
an elapsed time from a last checkpoint of a respective candidate job, calculated as a difference between a current time and the latest checkpoint time recorded in the first data structure for the respective candidate job and
a predicted remaining time for completion of the respective candidate job, calculated as a difference between the predicted total time for completion recorded in the second data structure for the respective candidate job,
the start time recorded in the second data structure for the respective candidate job, and
the current time, wherein the predicted total time for completion is predicted implicitly by applying job profiling based on resources needed and resources available to predict a total time for completion of the respective candidate job using extrapolation or one or more machine learning models;
ranking, by the one or more processors, the candidate jobs for preemption based on the dynamic information and the static information for each candidate job, wherein ranking the candidate jobs for preemption is done with an objective of minimizing loss of computation time while optimizing a benefit and utilization of resources of the computing cluster, and wherein ranking the candidate jobs for preemption comprises utilizing a unified metric of the dynamic information and the static information for scoring each candidate job, the unified metric being:
Mj=WCD*CDj+WLD*LDj+WPS*PS(j),
wherein WCD, WLD, and WPS are weights for aspect functions of candidate job j, wherein CD is a first function that produces a first score such that the first score is higher as the elapsed time from the last checkpoint is lower,
wherein LD is a second function that produces a second score such that the second score is higher as the predicted remaining time is higher, and
wherein PS is a third function that produces a third score such that the third score is higher as the configured priority value is lower;
attempting to preempt, by the one or more processors, a top N candidate jobs whose released resources in combination with the available resources of the computing cluster satisfy the required resources of the pending job; and
responsive to successfully preempting the top N candidate jobs, initiating, by the one or more processors, the pending job.
2 . The computer-implemented method of claim 1 , wherein the dynamic information is an elapsed time from a last checkpoint of a respective candidate job and a predicted remaining time for completion of the respective candidate job, and wherein the static information is a configured priority value assigned to the respective candidate job.
3 . The computer-implemented method of claim 1 , wherein determining that no computing hosts within the computing cluster have the available resources capable of running the pending job and the non-preemptive measures will not provide the required resources for the pending job comprises:
reviewing, by the one or more processors, current jobs running and the available resources within the computing cluster at a point in time by taking a snapshot; and
comparing, by the one or more processors, the available resources within the computing cluster to the required resources to run the pending job.
4 . The computer-implemented method of claim 1 , wherein identifying the static information for each candidate job comprises:
pulling, by the one or more processors, the static information for each candidate job from a job submission script.
5 . The computer-implemented method of claim 2 , wherein identifying the dynamic information for each candidate job comprises:
calculating, by the one or more processors, the elapsed time from the last checkpoint of the respective candidate job as a difference between a current time and a time of the last checkpoint recorded in a data structure for the respective candidate job, wherein the data structure stores a respective time of the last checkpoint detected for each running job within the computing cluster.
6 . The computer-implemented method of claim 5 , wherein the time of the last checkpoint is detected explicitly using a scheduler application programming interface (API) by which a running job marks successful completion of a checkpoint and records a time of completion of the checkpoint for the running job in the data structure.
7 . The computer-implemented method of claim 5 , wherein the time of the last checkpoint is detected implicitly by scanning logs of a running job for strings or patterns that indicate completion of a checkpoint.
8 . The computer-implemented method of claim 5 , wherein, in a container environment, the time of the last checkpoint is detected implicitly by monitoring volumes mounted to a container to detect changes in files that indicate completion of a checkpoint or by monitoring specific network traffic that would indicate completion of a snapshot.
9 . The computer-implemented method of claim 5 , wherein, for an artificial intelligence (AI) training job, the time of the last checkpoint is detected implicitly by monitoring contents of files that are specific for an AI training framework to detect completion of a checkpoint, wherein the AI training framework has specific identifiable files where calculation results are stored, progress is recorded, and stable checkpoints can be identified.
10 . The computer-implemented method of claim 2 , wherein identifying the dynamic information for each candidate job comprises:
calculating, by the one or more processors, the predicted remaining time for completion of the respective candidate job as a difference between a predicted total time for completion recorded in a data structure for the respective candidate job, a start time recorded in the data structure for the respective candidate job, and a current time, wherein the data structure stores a respective predicted total time for completion and a respective start time for each running job within the computing cluster.
11 . The computer-implemented method of claim 10 , wherein the predicted total time for completion is predicted explicitly using a scheduler application programming interface (API) by which a running job indicates the predicted total time for completion based on statistics.
12 . The computer-implemented method of claim 10 , wherein the predicted total time for completion is predicted implicitly by applying job profiling based on which resources are needed and which resources are available to predict a total time for completion of a job using extrapolation or machine learning models.
13 . The computer-implemented method of claim 2 , wherein identifying the dynamic information for each candidate job comprises:
predicting, by the one or more processors, the predicted remaining time for completion of the respective candidate job implicitly by using information on a current training stage and future training stages, wherein the respective candidate job is an artificial intelligence (AI) training job.
14 . The computer-implemented method of claim 1 , wherein ranking the candidate jobs for preemption based on the dynamic information and the static information for each candidate job is done with an objective of minimizing loss of computation time while optimizing a benefit and utilization of resources of the computer cluster.
15 . A computer program product comprising:
one or more computer-readable storage media and program instructions collectively stored on the one or more computer-readable storage media, the stored program instructions comprising:
program instructions to responsive to a pending job being selected to be run, determine that no computing hosts within a computing cluster have available resources capable of running the pending job and non-preemptive measures will not provide required resources for the pending job, wherein determining that no computing hosts within the computing cluster have available resources capable of running the pending job and the non-preemptive measures will not provide the required resources for the pending job comprises:
program instructions to review current jobs running and available resources within the computing cluster at a point in time by taking a snapshot; and program instructions to compare the available resources within the computing cluster to the required resources to run the pending job;
program instructions to identify candidate jobs for preemption and dynamic information and static information for each candidate job, wherein the static information is a configured priority value assigned to a respective candidate job and pulled from a job submission script, wherein the dynamic information is identified by polling the computing cluster for the dynamic information about each candidate job as that candidate job runs in the computing cluster, wherein the one or more processors maintain, in a job scheduler component:
(i) a first data structure that stores, for each running job within the computing cluster, a latest checkpoint time detected for that running job, the latest checkpoint time being recorded explicitly using a scheduler application programming interface (API) by which the running job marks successful completion of a checkpoint, and
(ii) a second data structure that stores, for each running job within the computing cluster, a predicted total time for completion and a start time for that running job, wherein the dynamic information includes:
an elapsed time from a last checkpoint of a respective candidate job, calculated as a difference between a current time and the latest checkpoint time recorded in the first data structure for the respective candidate job and
a predicted remaining time for completion of the respective candidate job, calculated as a difference between the predicted total time for completion recorded in the second data structure for the respective candidate job,
the start time recorded in the second data structure for the respective candidate job, and
the current time, wherein the predicted total time for completion is predicted implicitly by applying job profiling based on resources needed and resources available to predict a total time for completion of the respective candidate job using extrapolation or one or more machine learning models;
program instructions to rank the candidate jobs for preemption based on the dynamic information and the static information for each candidate job, wherein ranking the candidate jobs for preemption is done with an objective of minimizing loss of computation time while optimizing a benefit and utilization of resources of the computing cluster, and wherein ranking the candidate jobs for preemption comprises utilizing a unified metric of the dynamic information and the static information for scoring each candidate job, the unified metric being:
Mj=WCD*CDj+WLD*LDj+WPS*PS(j),
wherein WCD, WLD, and WPS are weights for aspect functions of candidate job j. wherein CD is a first function that produces a first score such that the first score is higher as the elapsed time from the last checkpoint is lower,
wherein LD is a second function that produces a second score such that the second score is higher as the predicted remaining time is higher, and
wherein PS is a third function that produces a third score such that the third score is higher as the configured priority value is lower;
program instructions to attempt to preempt a top N candidate jobs whose released resources in combination with the available resources of the computing cluster satisfy the required resources of the pending job; and
program instructions to responsive to successfully preempting the top N candidate jobs, initiate the pending job.
16 . The computer program product of claim 15 , wherein the dynamic information is an elapsed time from a last checkpoint of a respective candidate job and a predicted remaining time for completion of the respective candidate job, and wherein the static information is a configured priority value assigned to the respective candidate job.
17 . The computer program product of claim 16 , wherein the program instructions to identify the dynamic information for each candidate job comprises:
program instructions, stored on at least one of the one or more computer readable storage media, to calculate the elapsed time from the last checkpoint of the respective candidate job as a difference between a current time and a time of the last checkpoint recorded in a data structure for the respective candidate job, wherein the data structure stores a respective time of the last checkpoint detected for each running job within the computing cluster.
18 . The computer program product of claim 16 , wherein the program instructions to identify the dynamic information for each candidate job comprises:
program instructions, stored on at least one of the one or more computer readable storage media, to calculate the predicted remaining time for completion of the respective candidate job as a difference between a predicted total time for completion recorded in a data structure for the respective candidate job, a start time recorded in the data structure for the respective candidate job, and a current time, wherein the data structure stores a respective predicted total time for completion and a respective start time for each running job within the computing cluster.
19 . A computer system comprising:
one or more computer processors;
one or more computer-readable storage media;
program instructions collectively stored on the one or more computer-readable storage media for execution by at least one of the one or more computer processors, the stored program instructions comprising:
program instructions to responsive to a pending job being selected to be run, determine that no computing hosts within a computing cluster have available resources capable of running the pending job and non-preemptive measures will not provide required resources for the pending job, wherein determining that no computing hosts within the computing cluster have available resources capable of running the pending job and the non-preemptive measures will not provide the required resources for the pending job comprises:
program instructions to review current jobs running and available resources within the computing cluster at a point in time by taking a snapshot; and program instructions to compare the available resources within the computing cluster to the required resources to run the pending job;
program instructions to identify candidate jobs for preemption and dynamic information and static information for each candidate job, wherein the static information is a configured priority value assigned to a respective candidate job and pulled from a job submission script, wherein the dynamic information is identified by polling the computing cluster for the dynamic information about each candidate job as that candidate job runs in the computing cluster, wherein the one or more processors maintain, in a job scheduler component:
(i) a first data structure that stores, for each running job within the computing cluster, a latest checkpoint time detected for that running job, the latest checkpoint time being recorded explicitly using a scheduler application programming interface (API) by which the running job marks successful completion of a checkpoint, and
(ii) a second data structure that stores, for each running job within the computing cluster, a predicted total time for completion and a start time for that running job, wherein the dynamic information includes:
an elapsed time from a last checkpoint of a respective candidate job, calculated as a difference between a current time and the latest checkpoint time recorded in the first data structure for the respective candidate job and
a predicted remaining time for completion of the respective candidate job, calculated as a difference between the predicted total time for completion recorded in the second data structure for the respective candidate job,
the start time recorded in the second data structure for the respective candidate job, and
the current time, wherein the predicted total time for completion is predicted implicitly by applying job profiling based on resources needed and resources available to predict a total time for completion of the respective candidate job using extrapolation or one or more machine learning models;
program instructions to rank the candidate jobs for preemption based on the dynamic information and the static information for each candidate job, wherein ranking the candidate jobs for preemption is done with an objective of minimizing loss of computation time while optimizing a benefit and utilization of resources of the computing cluster, and wherein ranking the candidate jobs for preemption comprises utilizing a unified metric of the dynamic information and the static information for scoring each candidate job, the unified metric being:
Mj=WCD*CDj+WLD*LDj+WPS*PS(j),
wherein WCD, WLD, and WPS are weights for aspect functions of candidate job j, wherein CD is a first function that produces a first score such that the first score is higher as the elapsed time from the last checkpoint is lower,
wherein LD is a second function that produces a second score such that the second score is higher as the predicted remaining time is higher, and
wherein PS is a third function that produces a third score such that the third score is higher as the configured priority value is lower;
program instructions to attempt to preempt a top N candidate jobs whose released resources in combination with the available resources of the computing cluster satisfy the required resources of the pending job; and
program instructions to responsive to successfully preempting the top N candidate jobs, initiate the pending job.