Commanded message-based job flow cancellation in container-supported many task computing
An apparatus includes processor(s) to: receive, a request to cancel a performance of a job flow; store, within a task kill queue, a kill tasks request message; within a task container, in response to the storage of the kill tasks request message, cease execution of instructions of a task routine to cease performance of a task of a set of tasks of the job flow, and store, within a task queue, a task cancelation message indicating cancelation of execution of the task routine; within a performance container, execute instructions of a performance routine to cause the at least one processor to, in response to the storage of the task canceled message, store, within a job queue, a job cancelation message indicating cancelation of the job flow; and in response to the storage of the job cancelation message, transmit an indication of cancelation of the job flow to the requesting device.
1. An apparatus comprising at least one processor and a storage to store instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receive, at the at least one processor and from a requesting device via a network, a request to cancel a performance of a job flow, wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during the job flow performance;
the job flow definition is stored among multiple job flow definitions within at least one federated area;
the set of task routines is stored among multiple task routines within the at least one federated area; and
the at least one federated area is maintained within at least one storage device;
store, within a task kill queue, a kill tasks request message comprising a job flow identifier of the job flow;
within a first task container, in response to the storage of the kill tasks request message within the task kill queue, perform operations comprising:
cease execution of instructions of a first task routine of the set of task routines to cease performance of a first task of the set of tasks of the job flow; and
store, within a task queue, a first task cancelation message indicative of cancelation of execution of the first task routine;
within a second task container, in response to the storage of the kill tasks request message within the task kill queue, perform operations comprising:
cease execution of instructions of a second task routine of the set of task routines to cease performance a second task of the set of tasks of the job flow; and
store, within the task queue, a second task cancelation message indicative of cancelation of execution of the second task routine;
within a first performance container, execute instructions of a first instance of a performance routine to cause the at least one processor to, in response to the storage of either the first task canceled message within the task queue or the second task canceled message within the task queue, perform operations comprising:
store, within a job queue, a job cancelation message indicative of cancelation of the job flow; and
in response to the storage of the job cancelation message within the job queue, transmit an indication of cancelation of the job flow to the requesting device via the network.
2. The apparatus of claim 1 , wherein:
within the first performance container, a first task routine execution request message and a second task routine execution request message were stored within the task queue and the task queue was monitored for indications of completion of execution of the set of task routines in response to storage, within the job queue, of a job performance request message comprising the job flow definition of the job flow;
within the first task container, execution of the instructions of the first task routine commenced within the first task container in response to the storage of the first task routine execution request message within the task queue;
within the second task container, execution of the instructions of the second task routine commenced within the second task container in response to the storage of the second task routine execution request message within the task queue; and
the at least one processor is caused to, within the first performance container, in response to the storage of either the first task canceled message within the task queue or the second task canceled message within the task queue, perform operations comprising:
cease monitoring of the task queue for indications of completion of execution of the set of task routines.
3. The apparatus of claim 1 , wherein the at least one processor is caused to perform operations comprising:
within a first portal container, execute instructions of a first instance of a portal routine to cause the at least one processor to, in response to receiving the request to cancel the performance of the job flow from the requesting device, perform operations comprising:
in preparation for uninstantiation of the first portal container, store, within a job kill queue, a kill job request message comprising the job flow identifier of the job flow; and
in response to the storage of the kill job request message within the job kill queue:
generate the kill tasks request message;
store the kill tasks request message within the task kill queue;
monitor the job queue for the job cancelation message; and
in response to storage of the job cancelation message within the job queue, transmit the indication of cancelation of the job flow to the requesting device via the network.
4. The apparatus of claim 1 , wherein the at least one processor is caused to perform operations comprising:
within a first portal container, execute instructions of a first instance of a portal routine to cause the at least one processor to, in response to receiving the request to cancel the performance of the job flow from the requesting device, perform operations comprising:
in preparation for uninstantiation of the first portal container, store, within a job kill queue, a kill job request message comprising the job flow identifier of the job flow; and
in response to the storage of the kill job request message within the job kill queue, generate the kill tasks request message;
within a second portal container, execute instructions of a second instance of the portal routine to cause the at least one processor to, in response to the storage of the kill job request message within the job kill queue and in response to the uninstantiation of the first portal container, perform operations to take over for the first portal container that comprise:
generate the kill tasks request message;
store the kill tasks request message within the task kill queue;
monitor the job queue for the job cancelation message; and
in response to storage of the job cancelation message within the job queue, transmit the indication of cancelation of the job flow to the requesting device via the network.
5. The apparatus of claim 1 , wherein the at least one processor is caused to perform operations comprising:
receive, via the network, a request to perform the job flow;
retrieve the job flow definition from among the multiple job flow definitions in the at least one federated area;
store, within the job queue, a job performance request message comprising the job flow definition;
within the first performance container, in response to the storage of the job performance request message within the job queue, perform operations comprising:
derive an order of performance of the set of tasks that specifies at least the first task of the set of tasks to be performed, and that is based on indications in the job flow definition of dependencies among the set of tasks;
store, within the task queue, a first task routine execution request message comprising an identifier associated with the first task; and
monitor the task queue for an indication of completion of execution of the first task routine; and
within the first task container, in response to the storage of the first task routine execution request message within the task queue, perform operations comprising:
use the identifier associated with the first task to retrieve the first task routine from the at least one federated area; and
commence execution, within the first task container, of instructions of the first task routine to commence performance of the first task.
6. The apparatus of claim 5 , wherein:
the request to perform a job flow comprises the job flow identifier of the job flow definition;
the retrieval of the job flow definition comprises use of the job flow identifier, wherein the job flow definition comprises a set of flow task identifiers that each identify a task of the set of tasks;
the identifier associated with the first task comprises a flow task identifier of the set of flow task identifiers that identifies the first task; and
the retrieval of the first task routine comprises selecting, from among the multiple task routines stored within the at least one federated area, a latest version of task routine comprising instructions that, when executed by the at least processor, causes the at least one processor to perform the first task.
7. The apparatus of claim 5 , wherein:
the request to perform a job flow comprises a request to repeat a past performance of the job flow and includes an instance log identifier of an instance log that documents the past performance of the job flow, wherein the instance log comprises a set of task routine identifiers that each identify a task routine of the set of task routines;
the retrieval of the job flow definition comprises:
use of the instance log identifier to retrieve the instance log from among multiple instance logs stored within the at least one federated area;
retrieval of the job flow identifier of the job flow definition from the instance log; and
use of the job flow identifier to retrieve the job flow definition from the at least one federated area;
the identifier associated with the first task comprises a task routine identifier that identifies the first task; and
the retrieval of the first task routine comprises use of the task routine identifier to retrieve the first task routine from the at least one federated area.
8. The apparatus of claim 1 , wherein:
the at least one processor executes instructions of a resource allocation routine to cause the at least one processor to dynamically allocate multiple pods based on availability of at least one of processing resources and storage resources;
the first performance container and a first messaging container are instantiated within a first pod of the multiple pods;
within the first messaging container, instructions of a first instance of a messaging routine are executed by the at least one processor to cause the at least one processor to provide the first instance of the performance routine with access to the job queue and the task queue;
the first task container and a second messaging container are instantiated within a second pod of the multiple pods; and
within the second messaging container, instructions of a second instance of the messaging routine are executed by the at least one processor to cause the at least one processor to provide the first task routine with access to the task queue and to the task kill queue.
9. The apparatus of claim 8 , wherein:
a portal container and a third messaging container are instantiated within a third pod of the multiple pods;
within the portal container, the at least one processor executes instructions of an instance of a portal routine to cause the at least one processor to perform operations comprising:
receive the request to cancel the job flow from the requesting device;
determine whether the requesting device or an operator of the requesting device is authorized to request a cancellation of the performance of the job flow;
store the kill tasks request message within the task kill queue; and
in response to storage of the job cancelation message within the job queue, transmit the indication of cancelation of the job flow to the requesting device via the network.
10. The apparatus of claim 1 , wherein the at least one processor executes instructions of a message broker routine to provide multiple queues comprising the task kill queue, the task queue, the job queue and the job kill queue.
11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause at least one processor to perform operations comprising:
receive, at the at least one processor and from a requesting device via a network, a request to cancel a performance of a job flow, wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during the job flow performance;
the job flow definition is stored among multiple job flow definitions within at least one federated area;
the set of task routines is stored among multiple task routines within the at least one federated area; and
the at least one federated area is maintained within at least one storage device;
store, within a task kill queue, a kill tasks request message comprising a job flow identifier of the job flow;
within a first task container, in response to the storage of the kill tasks request message within the task kill queue, perform operations comprising:
cease execution of instructions of a first task routine of the set of task routines to cease performance of a first task of the set of tasks of the job flow; and
store, within a task queue, a first task cancelation message indicative of cancelation of execution of the first task routine;
within a second task container, in response to the storage of the kill tasks request message within the task kill queue, perform operations comprising:
cease execution of instructions of a second task routine of the set of task routines to cease performance a second task of the set of tasks of the job flow; and
store, within the task queue, a second task cancelation message indicative of cancelation of execution of the second task routine;
within a first performance container, execute instructions of a first instance of a performance routine to cause the at least one processor to, in response to the storage of either the first task canceled message within the task queue or the second task canceled message within the task queue, perform operations comprising:
store, within a job queue, a job cancelation message indicative of cancelation of the job flow; and
in response to the storage of the job cancelation message within the job queue, transmit an indication of cancelation of the job flow to the requesting device via the network.
12. The computer-program product of claim 11 , wherein:
within the first performance container, a first task routine execution request message and a second task routine execution request message were stored within the task queue and the task queue was monitored for indications of completion of execution of the set of task routines in response to storage, within the job queue, of a job performance request message comprising the job flow definition of the job flow;
within the first task container, execution of the instructions of the first task routine commenced within the first task container in response to the storage of the first task routine execution request message within the task queue;
within the second task container, execution of the instructions of the second task routine commenced within the second task container in response to the storage of the second task routine execution request message within the task queue;
and
the at least one processor is caused to, within the first performance container, in response to the storage of either the first task canceled message within the task queue or the second task canceled message within the task queue, perform operations comprising:
cease monitoring of the task queue for indications of completion of execution of the set of task routines.
13. The computer-program product of claim 11 , wherein the at least one processor is caused to perform operations comprising:
within a first portal container, execute instructions of a first instance of a portal routine to cause the at least one processor to, in response to receiving the request to cancel the performance of the job flow from the requesting device, perform operations comprising:
in preparation for uninstantiation of the first portal container, store, within a job kill queue, a kill job request message comprising the job flow identifier of the job flow; and
in response to the storage of the kill job request message within the job kill queue:
generate the kill tasks request message;
store the kill tasks request message within the task kill queue;
monitor the job queue for the job cancelation message; and
in response to storage of the job cancelation message within the job queue, transmit the indication of cancelation of the job flow to the requesting device via the network.
14. The computer-program product of claim 11 , wherein the at least one processor is caused to perform operations comprising:
within a first portal container, execute instructions of a first instance of a portal routine to cause the at least one processor to, in response to receiving the request to cancel the performance of the job flow from the requesting device, perform operations comprising:
in preparation for uninstantiation of the first portal container, store, within a job kill queue, a kill job request message comprising the job flow identifier of the job flow; and
in response to the storage of the kill job request message within the job kill queue, generate the kill tasks request message;
within a second portal container, execute instructions of a second instance of the portal routine to cause the at least one processor to, in response to the storage of the kill job request message within the job kill queue and in response to the uninstantiation of the first portal container, perform operations to take over for the first portal container that comprise:
generate the kill tasks request message;
store the kill tasks request message within the task kill queue;
monitor the job queue for the job cancelation message; and
in response to storage of the job cancelation message within the job queue, transmit the indication of cancelation of the job flow to the requesting device via the network.
15. The computer-program product of claim 11 , wherein the at least one processor is caused to perform operations comprising:
receive, via the network, a request to perform the job flow;
retrieve the job flow definition from among the multiple job flow definitions in the at least one federated area;
store, within the job queue, a job performance request message comprising the job flow definition;
within the first performance container, in response to the storage of the job performance request message within the job queue, perform operations comprising:
derive an order of performance of the set of tasks that specifies at least the first task of the set of tasks to be performed, and that is based on indications in the job flow definition of dependencies among the set of tasks;
store, within the task queue, a first task routine execution request message comprising an identifier associated with the first task; and
monitor the task queue for an indication of completion of execution of the first task routine; and
within the first task container, in response to the storage of the first task routine execution request message within the task queue, perform operations comprising:
use the identifier associated with the first task to retrieve the first task routine from the at least one federated area; and
commence execution, within the first task container, of instructions of the first task routine to commence performance of the first task.
16. The computer-program product of claim 15 , wherein:
the request to perform a job flow comprises the job flow identifier of the job flow definition;
the retrieval of the job flow definition comprises use of the job flow identifier, wherein the job flow definition comprises a set of flow task identifiers that each identify a task of the set of tasks;
the identifier associated with the first task comprises a flow task identifier of the set of flow task identifiers that identifies the first task; and
the retrieval of the first task routine comprises selecting, from among the multiple task routines stored within the at least one federated area, a latest version of task routine comprising instructions that, when executed by the at least processor, causes the at least one processor to perform the first task.
17. The computer-program product of claim 15 , wherein:
the request to perform a job flow comprises a request to repeat a past performance of the job flow and includes an instance log identifier of an instance log that documents the past performance of the job flow, wherein the instance log comprises a set of task routine identifiers that each identify a task routine of the set of task routines;
the retrieval of the job flow definition comprises:
use of the instance log identifier to retrieve the instance log from among multiple instance logs stored within the at least one federated area;
retrieval of the job flow identifier of the job flow definition from the instance log; and
use of the job flow identifier to retrieve the job flow definition from the at least one federated area;
the identifier associated with the first task comprises a task routine identifier that identifies the first task; and
the retrieval of the first task routine comprises use of the task routine identifier to retrieve the first task routine from the at least one federated area.
18. The computer-program product of claim 11 , wherein:
the at least one processor executes instructions of a resource allocation routine to cause the at least one processor to dynamically allocate multiple pods based on availability of at least one of processing resources and storage resources;
the first performance container and a first messaging container are instantiated within a first pod of the multiple pods;
within the first messaging container, instructions of a first instance of a messaging routine are executed by the at least one processor to cause the at least one processor to provide the first instance of the performance routine with access to the job queue and the task queue;
the first task container and a second messaging container are instantiated within a second pod of the multiple pods; and
within the second messaging container, instructions of a second instance of the messaging routine are executed by the at least one processor to cause the at least one processor to provide the first task routine with access to the task queue and to the task kill queue.
19. The computer-program product of claim 18 , wherein:
a portal container and a third messaging container are instantiated within a third pod of the multiple pods;
within the portal container, the at least one processor executes instructions of an instance of a portal routine to cause the at least one processor to perform operations comprising:
receive the request to cancel the job flow from the requesting device;
determine whether the requesting device or an operator of the requesting device is authorized to request a cancellation of the performance of the job flow;
store the kill tasks request message within the task kill queue; and
in response to storage of the job cancelation message within the job queue, transmit the indication of cancelation of the job flow to the requesting device via the network.
20. The computer-program product of claim 11 , wherein the at least one processor executes instructions of a message broker routine to provide multiple queues comprising the task kill queue, the task queue, the job queue and the job kill queue.
21. A method comprising:
receiving, by at the at least one processor and from a requesting device via a network, a request to cancel a performance of a job flow, wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during the job flow performance;
the job flow definition is stored among multiple job flow definitions within at least one federated area;
the set of task routines is stored among multiple task routines within the at least one federated area; and
the at least one federated area is maintained within at least one storage device;
storing, within a task kill queue, a kill tasks request message comprising a job flow identifier of the job flow;
within a first task container, in response to the storage of the kill tasks request message within the task kill queue, performing operations comprising:
ceasing execution of instructions of a first task routine of the set of task routines to cease performance of a first task of the set of tasks of the job flow; and
storing, within a task queue, a first task cancelation message indicative of cancelation of execution of the first task routine;
within a second task container, in response to the storage of the kill tasks request message within the task kill queue, performing operations comprising:
ceasing execution of instructions of a second task routine of the set of task routines to cease performance a second task of the set of tasks of the job flow; and
storing, within the task queue, a second task cancelation message indicative of cancelation of execution of the second task routine;
within a first performance container, executing instructions of a first instance of a performance routine causing the at least one processor to, in response to the storage of either the first task canceled message within the task queue or the second task canceled message within the task queue, perform operations comprising:
storing, within a job queue, a job cancelation message indicative of cancelation of the job flow; and
in response to the storage of the job cancelation message within the job queue, transmitting, from the at least one processor, an indication of cancelation of the job flow to the requesting device via the network.
22. The method of claim 21 , wherein:
within the first performance container, a first task routine execution request message and a second task routine execution request message were stored within the task queue and the task queue was monitored for indications of completion of execution of the set of task routines in response to storage, within the job queue, of a job performance request message comprising the job flow definition of the job flow;
within the first task container, execution of the instructions of the first task routine commenced within the first task container in response to the storage of the first task routine execution request message within the task queue;
within the second task container, execution of the instructions of the second task routine commenced within the second task container in response to the storage of the second task routine execution request message within the task queue;
and
the method comprising, within the first performance container, in response to the storage of either the first task canceled message within the task queue or the second task canceled message within the task queue, performing operations comprising:
ceasing monitoring of the task queue for indications of completion of execution of the set of task routines.
23. The method of claim 21 , comprising:
within a first portal container, executing instructions of a first instance of a portal routine causing the at least one processor to, in response to receiving the request to cancel the performance of the job flow from the requesting device, perform operations comprising:
in preparation for uninstantiation of the first portal container, storing, within a job kill queue, a kill job request message comprising the job flow identifier of the job flow; and
in response to the storage of the kill job request message within the job kill queue:
generating the kill tasks request message;
storing the kill tasks request message within the task kill queue;
monitoring the job queue for the job cancelation message; and
in response to storage of the job cancelation message within the job queue, transmitting the indication of cancelation of the job flow to the requesting device via the network.
24. The method of claim 21 , comprising:
within a first portal container, executing instructions of a first instance of a portal routine causing the at least one processor to, in response to receiving the request to cancel the performance of the job flow from the requesting device, perform operations comprising:
in preparation for uninstantiation of the first portal container, storing, within a job kill queue, a kill job request message comprising the job flow identifier of the job flow; and
in response to the storage of the kill job request message within the job kill queue, generating the kill tasks request message;
within a second portal container, executing instructions of a second instance of the portal routine causing the at least one processor to, in response to the storage of the kill job request message within the job kill queue and in response to the uninstantiation of the first portal container, perform operations to take over for the first portal container that comprise:
generating the kill tasks request message;
storing the kill tasks request message within the task kill queue;
monitoring the job queue for the job cancelation message; and
in response to storage of the job cancelation message within the job queue, transmitting the indication of cancelation of the job flow to the requesting device via the network.
25. The method of claim 21 , comprising:
receiving, via the network, a request to perform the job flow;
retrieving the job flow definition from among the multiple job flow definitions in the at least one federated area;
storing, within the job queue, a job performance request message comprising the job flow definition;
within the first performance container, in response to the storage of the job performance request message within the job queue, performing operations comprising:
deriving an order of performance of the set of tasks that specifies at least the first task of the set of tasks to be performed, and that is based on indications in the job flow definition of dependencies among the set of tasks;
storing, within the task queue, a first task routine execution request message comprising an identifier associated with the first task; and
monitoring the task queue for an indication of completion of execution of the first task routine; and
within the first task container, in response to the storage of the first task routine execution request message within the task queue, performing operations comprising:
using the identifier associated with the first task to retrieve the first task routine from the at least one federated area; and
commencing execution, within the first task container, of instructions of the first task routine to commence performance of the first task.
26. The method of claim 25 , wherein:
the request to perform a job flow comprises the job flow identifier of the job flow definition;
retrieving the job flow definition comprises using of the job flow identifier, wherein the job flow definition comprises a set of flow task identifiers that each identify a task of the set of tasks;
the identifier associated with the first task comprises a flow task identifier of the set of flow task identifiers that identifies the first task; and
retrieving the first task routine comprises selecting, from among the multiple task routines stored within the at least one federated area, a latest version of task routine comprising instructions that, when executed by the at least processor, causes the at least one processor to perform the first task.
27. The method of claim 25 , wherein:
the request to perform a job flow comprises a request to repeat a past performance of the job flow and includes an instance log identifier of an instance log that documents the past performance of the job flow, wherein the instance log comprises a set of task routine identifiers that each identify a task routine of the set of task routines;
retrieving the job flow definition comprises:
using of the instance log identifier to retrieve the instance log from among multiple instance logs stored within the at least one federated area;
retrieving the job flow identifier of the job flow definition from the instance log; and
using the job flow identifier to retrieve the job flow definition from the at least one federated area;
the identifier associated with the first task comprises a task routine identifier that identifies the first task; and
retrieving the first task routine comprises using of the task routine identifier to retrieve the first task routine from the at least one federated area.
28. The method of claim 21 , wherein:
the at least one processor executes instructions of a resource allocation routine to cause the at least one processor to dynamically allocate multiple pods based on availability of at least one of processing resources and storage resources;
the first performance container and a first messaging container are instantiated within a first pod of the multiple pods;
within the first messaging container, instructions of a first instance of a messaging routine are executed by the at least one processor to cause the at least one processor to provide the first instance of the performance routine with access to the job queue and the task queue;
the first task container and a second messaging container are instantiated within a second pod of the multiple pods; and
within the second messaging container, instructions of a second instance of the messaging routine are executed by the at least one processor to cause the at least one processor to provide the first task routine with access to the task queue and to the task kill queue.
29. The method of claim 28 , wherein:
a portal container and a third messaging container are instantiated within a third pod of the multiple pods;
within the portal container, the at least one processor executes instructions of an instance of a portal routine to cause the at least one processor to perform operations comprising:
receive the request to cancel the job flow from the requesting device;
determine whether the requesting device or an operator of the requesting device is authorized to request a cancellation of the performance of the job flow;
store the kill tasks request message within the task kill queue; and
in response to storage of the job cancelation message within the job queue, transmit the indication of cancelation of the job flow to the requesting device via the network.
30. The method of claim 21 , wherein the at least one processor executes instructions of a message broker routine to provide multiple queues comprising the task kill queue, the task queue, the job queue and the job kill queue.