System and method of providing system jobs within a compute environment
The invention relates to systems, methods and computer-readable media for using system jobs for performing actions outside the constraints of batch compute jobs submitted to a compute environment such as a cluster or a grid. The method for modifying a compute environment from a system job comprises associating a system job to a queuable object, triggering the system job based on an event and performing arbitrary actions on resources outside of compute nodes in the compute environment. The queuable objects include objects such as batch compute jobs or job reservations. The events that trigger the system job may be time driven, such as ten minutes prior to completion of the batch compute job, or dependent on other actions associated with other system jobs. The system jobs may be utilized also to perform rolling maintenance on a node by node basis.
1. A method for managing a multi-node compute environment, the method comprising:
associating a system job to a compute job that is scheduled according to a reservation to consume resources in a multi-node compute environment under common management; and
performing an action on a resource within the multi-node compute environment by processing the system job, the performed action being associated with actions other than (1) establishing any kind of reservation for resources to be consumed by the compute job and (2) directly dispatching and executing the compute job in the multi-node compute environment.
2. The method of claim 1 , wherein the system job has at least one attribute selected from a list comprising event type, timeout, offset, action, dependency, argument list, state and threshold.
3. The method of claim 2 , wherein the event type is at least one of a cancel, state and create event type.
4. The method of claim 1 , wherein performing the action is based on a time offset associated with the beginning or completion of the compute job.
5. The method of claim 1 , wherein performing the action comprising measuring a condition within the multi-node compute environment.
6. The method of claim 1 , further comprising, after performing the action:
performing a health check of resources affected by the performed action.
7. The method of claim 6 , wherein if the health check indicates that the resources affected by the performed action are operational, releasing the resources for use in the multi-node compute environment for batch compute jobs.
8. The method of claim 6 , wherein if the health check indicates that the resources affected by the performed action is not operational, sending a message to an administrator.
9. The method of claim 1 , wherein performing the action is triggered by an event associated with a second system job.
10. The method of claim 9 , wherein the event associated with the second system job is the completion of the second system job.
11. The method of claim 1 , wherein performing the action is dependent on a second system job.
12. A system for managing a multi-node compute environment, the system comprising:
a processor;
a first module the processor to associate a system job to a compute job that is scheduled according to a reservation to consume resources in a multi-node compute environment; and
a second module controlling the processor to perform an action on a resource within the multi-node compute environment by processing the system job, the performed action being associated with actions other than (1) establishing any kind of reservation for resources to be consumed by the compute job and (2) directly dispatching and executing the compute job in the multi-node compute environment.
13. The system of claim 12 , wherein the second module further performs the action based on a time offset associated with a beginning or a completion of the compute job.
14. A non-transitory computer-readable storage medium storing instructions for managing a multi-node compute environment, the instructions comprising:
associating a system job to a compute job that is scheduled according to a reservation to consume resources in a multi-node compute environment; and
performing an action on a resource within the multi-node compute environment by processing the system job, the performed action being associated with actions other than (1) establishing any kind of reservation for resources to be consumed by the compute job and (2) directly dispatching and executing the compute job in the multi-node compute environment.
15. The non-transitory computer-readable storage medium of claim 14 , wherein the instruction that performs the action is based on a time offset associated with the beginning or the completion of the compute job.
16. The non-transitory computer-readable storage medium of claim 14 , wherein performing the action is triggered by an event associated with a second system job.