Automated generation of scheduling algorithms based on task relevance assessment
A method for automatically generating scheduling algorithms, including determining a scheduling policy for a plurality of jobs to be executed on a computer system, where the scheduling policy specifies an execution order of a plurality of jobs; using the scheduling policy in a production environment for a period of time, and collecting data indicative of a business impact of each job executed during the period of time; generating a list of all pairwise comparisons of business impact between the plurality of jobs, together with outcomes of the comparisons; marking each pair for which the comparison outcome is inconsistent with the relative execution order of the pair of jobs according to a predefined criteria to create a reinforcement learning batch; and using the reinforcement learning batch to adjust a decision criteria used to determine the scheduling policy.
1. A method for automatically generating a scheduling algorithm for a computer system, comprising the steps of:
defining a set of features for each job of a plurality of jobs;
defining a set of priority classes into which the plurality of jobs are classified;
clustering a set of historic job data to define clusters of jobs belonging to a same priority class, and identifying a representative job for each cluster, wherein the representative job for each cluster is based on a centroid of each cluster;
defining a priority level for each cluster;
executing incoming jobs on said computer system based on job requests received from users;
identifying a cluster to which each incoming job belongs by computing an Euclidean distance between each incoming job and the representative job of each cluster and selecting the cluster associated with a shortest distance;
calculating an average of the shortest distances between each of the most recent number of incoming jobs and the representative job of each incoming job's respective cluster;
determining whether said average distance exceeds surpasses a pre-defined threshold, and
defining a new set of priority classes into which the plurality of jobs are classified, when said average distance exceeds surpasses said pre-defined threshold.
2. The method of claim 1 , wherein priorities are assigned to clusters based on a business value of the jobs in the cluster, wherein jobs in more valuable clusters receive a higher priority.
3. The method of claim 1 , further comprising storing the distance between each incoming job and the representative of its cluster.
4. The method of claim 1 , wherein executing jobs on said computer system comprises:
receiving job requests;
extracting features from each received job;
assigning a priority to the received job based on the cluster to which said job belongs; submitting the job and its priority level to a scheduler; and
executing said job.
5. A non-transitory program storage device readable by a computer, tangibly embodying a program of instructions executed by the computer to perform the method steps for automatically generating a scheduling algorithm for a computer system, comprising the steps of:
defining a set of features for each job of a plurality of jobs;
defining a set of priority classes into which the plurality of jobs are classified;
clustering a set of historic job data to define clusters of jobs belonging to a same priority class, and identifying a representative job for each cluster, wherein the representative job for each cluster is based on a centroid of each cluster;
defining a priority level for each cluster;
executing incoming jobs on said computer system based on job requests received from users;
identifying a cluster to which each incoming job belongs by computing an Euclidean distance between each incoming job and the representative job of each cluster and selecting the cluster associated with a shortest distance;
calculating an average of the shortest distances between each of the most recent number of incoming jobs and the representative job of each incoming job's respective cluster;
determining whether said average distance exceeds surpasses a pre-defined threshold, and
defining a new set of priority classes into which the plurality of jobs are classified, when said average distance exceeds surpasses said pre-defined threshold.
6. The computer readable program storage device of claim 5 , wherein priorities are assigned to clusters based on a business value of the jobs in the cluster, wherein jobs in more valuable clusters receive a higher priority.
7. The computer readable program storage device of claim 5 , wherein the method further comprises storing the distance between an each incoming job and the representative of its cluster.
8. The computer readable program storage device of claim 5 , wherein executing jobs on said computer system comprises:
receiving job requests;
extracting features from each received job;
assigning a priority to the received job based on the cluster to which said job belongs; submitting the job and its priority level to a scheduler; and
executing said job.