System and method of providing system jobs within a compute environment
View Patent ↗The disclosure relates to systems, methods and computer-readable media for using system jobs for performing actions outside the constraints of batch compute jobs submitted to a compute environment such as a cluster or a grid. The method for modifying a compute environment from a system job disclosure associating a system job to a queuable object, triggering the system job based on an event and performing arbitrary actions on resources outside of compute nodes in the compute environment. The queuable objects include objects such as batch compute jobs or job reservations. The events that trigger the system job may be time driven, such as ten minutes prior to completion of the batch compute job, or dependent on other actions associated with other system jobs. The system jobs may be utilized also to perform rolling maintenance on a node by node basis.
1. A method of performing rolling maintenance on a node within a compute environment, the method comprising:
receiving a submission of a system job associated with a node;
identifying an earliest possible time for an update to the node, wherein the update comprises one or more of reinstalling software or updating software on the node;
automatically performing, at a predetermined time, the update on the node;
determining whether the updating was successful, and if the updating was successful, terminating the system job leaving the node available for use in the compute environment;
if the updating was not successful, communicating an unsuccessful status report and creating a reservation for the node.
2. The method of claim 1 , wherein the updating operation includes updating an operating system.
3. The method of claim 1 , wherein the updating operation on each node is performed and on a node-by-node basis independently.
4. The method of claim 1 , wherein the predetermined time is set to be one of the earliest possible time, a scheduled time or an earliest possible time after a predetermined period of time.
5. The method of claim 1 , wherein system jobs are set to be submitted at both a grid level and a cluster level within a compute environment.
6. The method of claim 1 , further comprising associating the system job to a queueable object.
7. The method of claim 6 , wherein performing the system job is based on a time offset associated with one of a beginning of or a completion of the queueable object.
8. The method of claim 1 , wherein communicating an unsuccessful status report comprises communicating a message to an administrator.
9. A system for performing rolling maintenance on a node within a compute environment, the system comprising:
a processor;
a computer-readable storage medium storing instructions which, when executed by the processor, cause the processor to perform operations comprising:
receiving a submission of a system job associated with a node;
identifying an earliest possible time for an update to the node, wherein the update comprises one or more of installing software or updating software on the node;
automatically performing, at a predetermined time the update on the node;
determining whether the updating was successful and if the updating was successful, terminating the system job leaving the node available for use in the compute environment;
if the updating was not successful, communicating an unsuccessful status report and creating a reservation for the node.
10. The system of claim 9 , wherein the updating operation on each node is performed independently and on a node-by-node basis.
11. The system of claim 9 , wherein the predetermined time is set to be one of the earliest possible time, a scheduled time, or an earliest possible time after a predetermined period of time.
12. The system of claim 9 , wherein system jobs are set to be submitted at both a grid level and a cluster level within a compute environment.
13. The system of claim 9 , wherein the processor is further configured to associate the system job to a queueable object.
14. The system of claim 13 , wherein performing the system job is based on a time offset associated with one of a beginning of or a completion of the queueable object.
15. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform operations comprising:
receiving a submission of a system job associated with a node;
identifying an earliest possible time for an update to the node, wherein the update comprises one or more of installing software or updating software on the node;
automatically performing, at a predetermined time the update on the node;
determining whether the updating was successful and if the updating was successful, terminating the system job leaving the node available for use in the compute environment;
if the updating was not successful, communicating an unsuccessful status report and creating a reservation for the node.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the predetermined time operation is set to be one of the earliest possible time, a scheduled time or an earliest possible time after a predetermined period of time.
17. The non-transitory computer-readable storage medium of claim 15 , wherein system jobs are set to be submitted at both a grid level and a cluster level within a compute environment.
18. The non-transitory computer-readable storage medium of claim 15 , wherein the processor is further configured to associate the system job to a queueable object.
19. The non-transitory computer-readable storage medium of claim 18 , wherein performing the system job is based on a time offset associated with one of a beginning of or a completion of the queueable object.