System and method of providing system jobs within a compute environment
The disclosure relates to systems, methods and computer-readable media for using system jobs for performing actions outside the constraints of batch compute jobs submitted to a compute environment such as a cluster or a grid. The method for modifying a compute environment from a system job disclosure associating a system job to a queuable object, triggering the system job based on an event and performing arbitrary actions on resources outside of compute nodes in the compute environment. The queuable objects include objects such as batch compute jobs or job reservations. The events that trigger the system job may be time driven, such as ten minutes prior to completion of the batch compute job, or dependent on other actions associated with other system jobs. The system jobs may be utilized also to perform rolling maintenance on a node by node basis.
1. A non-transitory computer-readable medium storing instructions for managing a multi-node compute environment having a computerized scheduler process associated therewith, the instructions, when executed by a processor of a computerized device, performs operations comprising:
receiving a submission of at least one workload to be processed by the multi-node compute environment, the submission also comprising data indicating a particular quality of service (QoS) associated with the processing of the at least one workload;
creating at least one software process associated with the at least one workload;
determining that at least one resource necessary for processing of the at least one workload according to the particular QoS has been made available for processing of the at least one workload; and
performing via the at least one software process and based on the determination, one or more configuration actions that are outside of a compute space allocated by the computerized scheduler process,
wherein:
the at least one software process comprises an event policy relating to occurrence of one or more events, and the completion of the one or more configuration actions comprises the occurrence of the one or more events;
the one or more configuration actions comprise one or more provisioning actions related to the at least one resource; the data indicating a particular quality of service (QoS) associated with the processing of the at least one workload comprises data indicative of a requirement for at least one dedicated resource; and
the at least one dedicated resource comprises the at least one resource.
2. The non-transitory computer-readable medium of claim 1 , wherein the at least one software process comprises an event policy relating to occurrence of one or more events, and completion of the one or more configuration actions comprises the occurrence of the one or more events.
3. The non-transitory computer-readable medium of claim 1 , wherein the at least one software process comprises an event policy relating to occurrence of one or more events, and the determination that the at least one resource has been made available for processing of the at least one workload comprises the occurrence of the one or more events.
4. The non-transitory computer-readable medium of claim 1 , wherein the one or more configuration actions comprise one or more provisioning actions related to the at least one resource.
5. The non-transitory computer-readable medium of claim 1 , wherein the data indicating a particular quality of service (QoS) associated with the processing of the at least one workload comprises data indicative of a requirement for at least one dedicated resource.
6. The non-transitory computer-readable medium of claim 1 , wherein:
the at least one software process comprises a system job; and
at least the creation of the system job is performed by the computerized scheduler process.
7. The non-transitory computer-readable medium of claim 1 , wherein:
the at least one software process comprises a system job; and
the association of the system job with the at least one workload comprises at least an encapsulation of at least one of (i) at least a portion of the submission, or (ii) at least a portion of the at least one workload, within the system job.
8. The non-transitory computer-readable medium of claim 1 , wherein:
the at least one software process comprises a system job; and
the performance of the one or more configuration actions that are outside of the compute space allocated by the computerized scheduler process comprises the performance of a plurality of steps having at least one temporal time constraint relating to performance or completion of at least one of the plurality of steps.
9. The non-transitory computer-readable medium of claim 1 , wherein:
the at least one software process comprises at least one queuable object; and
the at least one queueable object is queued for processing by the computerized scheduler process.
10. A non-transitory computer-readable medium storing instructions for managing a multi-node compute environment having a computerized scheduler process associated therewith, the instructions, when executed by a processor of a computerized device, performs operations comprising:
receiving a submission of at least one workload to be processed by the multi-node compute environment, the submission requiring a particular quality of service (QoS) for the processing of the at least one workload;
configuring, via at least one software process associated with the at least one workload, at least one resource necessary for processing the at least one workload according to the particular QoS; and
based at least on completion of the configuring of the at least one resource, performing via the at least one software process, at least part of the processing of the at least one workload,
wherein:
the at least one software process comprises an event policy relating to occurrence of one or more events, and the completion of the configuration comprises the occurrence of the one or more events;
the configuring comprises one or more provisioning actions related to the at least one resource; the particular quality of service (QoS) for processing of the at least one workload comprises a requirement for at least one dedicated resource; and
the at least one dedicated resource comprises the at least one resource.
11. The non-transitory computer-readable medium of claim 10 , wherein the configuring comprises one or more actions not able to be performed by the computerized scheduler process.
12. The non-transitory computer-readable medium of claim 10 , wherein the at least one software process comprises an event policy relating to occurrence of one or more events, and completion of the configuring of the at least one resource comprises the occurrence of the one or more events.
13. The non-transitory computer-readable medium of claim 10 , wherein the at least one software process comprises an event policy relating to occurrence of one or more events, and a determination that the at least one resource has been made available for processing the at least one workload comprises the occurrence of the one or more events.
14. The non-transitory computer-readable medium of claim 10 , wherein the configuring comprises provisioning of data storage resources associated with one or more nodes of the multi-node compute environment.
15. The non-transitory computer-readable medium of claim 10 , wherein the particular quality of service (QoS) for the processing of the at least one workload comprises a requirement for at least one dedicated resource.
16. The non-transitory computer-readable medium of claim 10 , wherein:
the at least one software process comprises a system job; and
at least the creation of the system job is performed by the computerized scheduler process.
17. The non-transitory computer-readable medium of claim 10 , wherein:
the at least one software process comprises a system job; and
the association of the system job with the at least one workload comprises at least an encapsulation of at least one of (i) at least a portion of the submission, or (ii) at least a portion of the at least one workload, within the system job.
18. The non-transitory computer-readable medium of claim 10 , wherein:
the at least one software process comprises a system job; and
the configuring comprises the performance of a plurality of steps having at least one temporal time constraint relating to performance or completion of at least one of the plurality of steps.
19. The non-transitory computer-readable medium of claim 10 , wherein:
the at least one software process comprises at least one queuable object; and
the at least one queueable object is queued for processing by the computerized scheduler process.
20. A non-transitory computer-readable medium storing instructions for managing a multi-node compute environment having a computerized scheduler process associated therewith, the instructions, when executed by a processor of a computerized device, performs operations comprising:
creating at least one software process, the at least one software process configured to be associated with one or more queueable objects operative within the multi-node compute environment, the at least one software process comprising an event policy;
monitoring of at least one operational aspect of the multi-node compute environment; and
performing via the at least one software process and based on detecting at least one triggering event associated with the event policy, one or more file system backup actions,
wherein the at least one software object is configured to cause scheduling of execution of one or more configuration actions on at least one node of the multi-node compute environment, and
wherein the one or more configuration actions comprise performance of one or more configuration adjustments to the computerized scheduler process.
21. The non-transitory computer-readable medium of claim 20 , wherein:
the allocation of at least one resource associated with the file system is responsive to a feature requirement specified by at least one of (i) a user submitting compute workload for processing on by the multi-node compute environment; or (ii) the compute workload itself.