System and method of providing system jobs within a compute environment
View Patent ↗The disclosure relates to systems, methods and computer-readable media for using system jobs for performing actions outside the constraints of batch compute jobs submitted to a compute environment such as a cluster or a grid. The method for modifying a compute environment from a system job disclosure associating a system job to a queuable object, triggering the system job based on an event and performing arbitrary actions on resources outside of compute nodes in the compute environment. The queuable objects include objects such as batch compute jobs or job reservations. The events that trigger the system job may be time driven, such as ten minutes prior to completion of the batch compute job, or dependent on other actions associated with other system jobs. The system jobs may be utilized also to perform rolling maintenance on a node by node basis.
1. A method comprising:
associating a system job to a queuable object comprising one of a batch compute workload, a compute workload and a reservation in a multi-node compute environment under common management; and
performing an action on a resource within the multi-node compute environment by processing the system job to yield a performed action, the performed action being one action of a set of actions that are outside of actions managed by a workload manager or scheduler.
2. The method of claim 1 , wherein the set of actions comprises and event type, a timeout, an offset, an action, a dependency, an argument list, a state and a threshold.
3. The method of claim 1 , wherein the event type is at least one of a cancel, a state and a create event type.
4. The method of claim 1 , wherein performing the action is based on a time offset associated with a beginning of or a completion of the compute workload.
5. The method of claim 1 , wherein performing the action comprising measuring a condition within the multi-node compute environment.
6. The method of claim 1 , further comprising, after performing the action:
performing a health check of resources affected by the performed action.
7. The method of claim 6 , wherein if the health check indicates that the resources affected by the performed action are operational, releasing the resources for use in the multi-node compute environment for batch compute workload.
8. The method of claim 7 , wherein if the health check indicates that the resources affected by the performed action is not operational, sending a message to an administrator.
9. The method of claim 1 , wherein performing the action is triggered by an event associated with a second system job.
10. The method of claim 9 , wherein the event associated with the second system job is a completion of the second system job.
11. The method of claim 1 , wherein performing the action is dependent on a second system job.
12. The method of claim 1 , wherein actions managed by the workload manager or scheduler comprise at least reserving resources to be consumed by the queuable object and directly dispatching and executing the queuable object in the multi-node compute environment.
13. A system for managing a multi-node compute environment, the system comprising:
a processor; and
a computer-readable storage medium storing instructions which, when executed by the processor, cause the processor to perform operations comprising:
associating a system job to a queuable object comprising one of a batch compute workload, a compute workload and a reservation in a multi-node compute environment under common management; and
performing an action on a resource within the multi-node compute environment by processing the system job to yield a performed action, the performed action being one action of a set of actions that are outside of actions managed by a workload manager or scheduler.
14. The system of claim 13 , wherein performing the action is based on a time offset associated with a beginning or a completion of the queuable object.
15. A non-transitory computer-readable storage medium storing instructions which, when executed by the processor, cause the processor to perform operations comprising:
associating a system job to a queuable object comprising one of a batch compute workload, a compute workload and a reservation in a multi-node compute environment under common management; and
performing an action on a resource within the multi-node compute environment by processing the system job to yield a performed action, the performed action being one action of a set of actions that are outside of actions managed by a workload manager or scheduler.
16. The non-transitory computer-readable storage medium of claim 15 , wherein performing the action is based on a time offset associated with a beginning of or a completion of the queuable object.
17. The non-transitory computer-readable storage medium of claim 15 , wherein performing the action is triggered by an event associated with a second system job.
18. The non-transitory computer-readable storage medium of claim 15 , wherein the queuable object is compute workload.
19. The non-transitory computer-readable storage medium of claim 17 , wherein the event associated with the second system job is a completion of the second system job.
20. The non-transitory computer-readable storage medium of claim 15 , wherein actions managed by the workload manager or scheduler comprise at least reserving resources to be consumed by the queuable object and directly dispatching and executing the queuable object in the multi-node compute environment.