System and method of providing system jobs within a compute environment
View Patent ↗The invention relates to systems, methods and computer-readable media for using system jobs for performing actions outside the constraints of batch compute jobs submitted to a compute environment such as a cluster or a grid. The method for modifying a compute environment from a system job comprises associating a system job to a queuable object, triggering the system job based on an event and performing arbitrary actions on resources outside of compute nodes in the compute environment. The queuable objects include objects such as batch compute jobs or job reservations. The events that trigger the system job may be time driven, such as ten minutes prior to completion of the batch compute job, or dependent on other actions associated with other system jobs. The system jobs may be utilized also to perform rolling maintenance on a node by node basis.
1. A method comprising:
generating a system job in a multi-node compute environment, the system job differing from compute workload submitted by a user; and
performing an action on a resource within the multi-node compute environment by processing the system job to yield a performed action, the performed action being other than (1) establishing any kind of reservation for resources to be consumed by the compute workload and (2) directly dispatching and executing the compute workload in the multi-node compute environment.
2. The method of claim 1 , wherein the system job has at least one attribute selected from a list comprising event type, timeout, offset, action, dependency, argument list, state and threshold.
3. The method of claim 2 , wherein the event type is at least one of a cancel, state and create event type.
4. The method of claim 1 , wherein performing the action is based on a time offset associated with a beginning of or a completion of the compute workload.
5. The method of claim 1 , wherein performing the action comprising measuring a condition within the multi-node compute environment.
6. The method of claim 1 , further comprising, after performing the action:
performing a health check of resources affected by the performed action.
7. The method of claim 6 , wherein if the health check indicates that the resources affected by the performed action are operational, releasing the resources for use in the multi-node compute environment for batch compute workload.
8. The method of claim 6 , wherein if the health check indicates that the resources affected by the performed action is not operational, sending a message to an administrator.
9. The method of claim 1 , wherein performing the action is triggered by an event associated with a second system job.
10. The method of claim 9 , wherein the event associated with the second system job is a completion of the second system job.
11. The method of claim 1 , wherein performing the action is dependent on a second system job.
12. A system for managing a multi-node compute environment, the system comprising:
a processor; and
a computer-readable storage medium storing instructions, which, when executed by the processor, cause the processor to perform operations comprising:
generating a system job in a multi-node compute environment, the system job differing from a compute workload; and
performing an action on a resource within the multi-node compute environment by processing the system job to yield a performed action, the performed action being other than (1) establishing any kind of reservation for resources to be consumed by the compute workload and (2) directly dispatching and executing the compute workload in the multi-node compute environment.
13. The system of claim 12 , wherein the computer-readable medium stores instructions which, when executed by the processor, perform operations further comprising:
performing the action based on a time offset associated with a beginning of or a completion of the compute workload.
14. A computer-readable storage medium storing instructions for managing a multi-node compute environment, the instructions causing a computing device to perform operations comprising:
associating a system job to a compute workload that is scheduled to consume resources in a multi-node compute environment; and
performing an action on a resource within the multi-node compute environment by processing the system job to yield a performed action, the performed action being other than (1) establishing any kind of reservation for resources to be consumed by the compute workload and (2) directly dispatching and executing the compute workload in the multi-node compute environment.
15. The computer-readable storage medium of claim 14 , wherein performing the action is based on a time offset associated with a beginning or a completion of the compute workload.
16. The computer-readable storage medium of claim 14 , wherein performing the action is triggered by an event associated with a second system job.