Granted Patent B2

US 12,333,343 · App. 17/456,210 · Granted Jun 17, 2025

Inventors: Hai Hui Wang (Xian, CN); Shan Gao (Beijing, CN); Yang Gao (Xian, CN)

Assignee: International Business Machines Corporation

G06F9/505G06F9/542

Abstract

Claims (64)

1. A method for avoiding workload duplication in a cluster environment, the method comprising:

identifying, by a computer system, a state change among a set of cluster resources in a cluster of nodes;

responsive to identifying the state change, predicting, by the computer system, resource requirements for a queued workload;

determining, by the computer system, a pre-assignment of the queued workload to a sub-cluster according to the resource requirements that were predicted for the queued workload;

marking, by the computer system, the queued workload to indicate the pre-assignment to the sub-cluster; and

scheduling, by the sub-cluster of the computer system, the queued workload for execution on the sub-cluster, wherein the sub-cluster does not schedule queued workloads that are pre-assigned to other sub-clusters.

2. The method of claim 1 , wherein the steps of predicting resource requirements, determining the pre-assignment, and marking the queued workload are provided as a set of dispatch policies that are automatically enabled when the state change is identified among the set of cluster resources.

3. The method of claim 2 wherein the set of dispatch policies comprises custom and built-in policies that provide configurable rules for distinguishing the sub-cluster from other sub-cluster s.

4. The method of claim 1 , further comprising:

responsive to a failure of a network connection between the cluster of nodes, scheduling, by the computer system, the queued workload according to the pre-assignment.

5. The method of claim 4 , wherein the step of scheduling the queued workload further comprises:

identifying, by the sub-cluster of the computer system, the pre-assignment of the queued workload to the sub-cluster.

6. The method of claim 1 , wherein predicting the resource requirements further comprises:

modeling, by the computer system, the queued workload on a set of machine learning models trained on the resource requirements of a completed workload; and

predicting, by the computer system, the resource requirements of the queued workload from the set of machine learning models.

7. The method of claim 6 , wherein modeling the queued workload further comprises:

generating, by the computer system, metadata about the resource requirements of the completed workload;

collecting, by the computer system, the metadata into a training data set; and

training, by the computer system, the set of machine learning models from the training data set.

8. The method of claim 7 , wherein the metadata comprises:

a pending time of the completed workload;

a running time of the completed workload; and

a resource usage of the completed workload.

9. A computer system comprising:

a number of storage devices that store program instructions; and

a number of processor units in communication with the number of storage devices, wherein the number of processor units executes program instructions to:

identify a state change among a set of cluster resources in a cluster of nodes;

predict, in response to identifying the state change, resource requirements for a queued workload;

determine a pre-assignment of the queued workload to a sub-cluster according to the resource requirements that were predicted for the queued workload;

mark the queued workload to indicate the pre-assignment to the sub-cluster; and

scheduling, by the sub-cluster of the computer system, the queued workload for execution on the sub-cluster, wherein the sub-cluster does not schedule queued workloads that are pre-assigned to other sub-clusters.

10. The computer system of claim 9 , wherein the steps of predicting resource requirements, determining the pre-assignment, and marking the queued workload are provided as a set of dispatch policies that are automatically enabled when the state change is identified among the set of cluster resources.

11. The computer system of claim 10 , wherein the set of dispatch policies comprises custom and built-in policies that provide configurable rules for distinguishing the sub-cluster from other sub-clusters.

12. The computer system of claim 9 , wherein the processor further executes the program instructions to:

schedule, responsive to a failure of a network connection between the cluster of nodes, the queued workload according to the pre-assignment.

13. The computer system of claim 12 , wherein in scheduling the queued workload, the processor further executes the program instructions:

to identify the pre-assignment of the queued workload to the sub-cluster.

14. The computer system of claim 9 , wherein in predicting the resource requirements, the processor further executes the program instructions to:

model the queued workload on a set of machine learning models trained on the resource requirements of a completed workload; and

predict the resource requirements of the queued workload from the set of machine learning models.

15. The computer system of claim 14 , wherein in modeling the queued workload, the processor further executes the program instructions to:

generate metadata about the resource requirements of the completed workload;

collect the metadata into a training data set; and

train the set of machine learning models from the training data set.

16. The computer system of claim 15 , wherein the metadata comprises:

a pending time of the completed workload;

a running time of the completed workload; and

a resource usage of the completed workload.

17. A computer program product for avoiding workload duplication in a cluster, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer system to cause the computer system to perform a method of:

identifying a state change among a set of cluster resources in a cluster of nodes;

responsive to identifying the state change, predicting resource requirements for a queued workload;

determining a pre-assignment of the queued workload to a sub-cluster according to the resource requirements that were predicted for the queued workload;

marking the queued workload to indicate the pre-assignment to the sub-cluster; and

scheduling, by the sub-cluster of the computer system, the queued workload for execution on the sub-cluster, wherein the sub-cluster does not schedule queued workloads that are pre-assigned to other sub-clusters.

18. The computer program product of claim 17 , further comprising:

responsive to a failure of a network connection between the cluster of nodes, scheduling the queued workload according to the pre-assignment, wherein scheduling the queued workload further comprises:

identifying, by the sub-cluster, the pre-assignment of the queued workload to the sub-cluster.

19. The computer program product of claim 17 , wherein predicting the resource requirements further comprises:

modeling the queued workload on a set of machine learning models trained on the resource requirements of a completed workload; and

predicting the resource requirements of the queued workload from the set of machine learning models.

20. The computer program product of claim 19 , wherein modeling the queued workload further comprises:

generating metadata about the resource requirements of the completed workload;

collecting the metadata into a training data set; and

training the set of machine learning models from the training data set.

Assignments (1)

Continuity (1)

References Cited (23)