Message-based coordination of container-supported many task computing
An apparatus includes processor(s) to: receive a request to perform a job flow; retrieve a job flow definition from federated area(s); store, within a job queue, a job performance request message comprising the job flow definition; within a first performance container, execute a performance routine causing the processor(s) to, in response to the storage of the job performance request message within the job queue, derive an order of performance of tasks specifying a first task based on dependencies among the tasks, and store, within a task queue, a first task routine execution request message comprising an identifier associated with the first task; and within a first task container, in response to the storage of the first task routine execution request message within the task queue, use the identifier to retrieve a first task routine from the federated area(s), and execute the first task routine to perform the first task.
1. An apparatus comprising at least one processor and a storage to store instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receive, at the at least one processor and from a requesting device via a network, a request to perform a job flow, wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during the job flow performance;
the job flow definition is stored among multiple job flow definitions within at least one federated area;
the set of task routines is stored among multiple task routines within the at least one federated area; and
the at least one federated area is maintained within at least one storage device;
retrieve the job flow definition from among the multiple job flow definitions in the at least one federated area;
store, within a job queue, a job performance request message comprising the job flow definition;
within a first performance container, execute instructions of a first instance of a performance routine to cause the at least one processor to, in response to the storage of the job performance request message within the job queue, perform operations comprising:
derive an order of performance of the set of tasks that specifies at least a first task of the set of tasks to be performed and a second task to be performed after the first task, and that is based on indications in the job flow definition of dependencies among the set of tasks;
store, within a task queue, a first task routine execution request message comprising an identifier associated with the first task; and
in response to at least storage of an execution completion message indicative of completion of execution of the first task routine within the task queue, store, within the task queue, a second task routine execution request message comprising an identifier associated with the second task to cause a latest version of a second task routine to be executed within a second task container to cause the at least one processor to perform the second task; and
within a first task container, in response to the storage of the first task routine execution request message within the task queue, perform operations comprising:
use the identifier associated with the first task to retrieve a latest version of a first task routine of the set of task routines from the at least one federated area;
commence execution, within the first task container, of instructions of the first task routine to commence performance of the first task; and
following completion of the execution of the first task routine, store, within the task queue, the execution completion message indicative of completion of execution of the first task routine.
2. The apparatus of claim 1 , wherein the at least one processor is caused to perform operations comprising:
within the first task container, following completion of the execution of the instructions of the first task routine, store, within the task queue, an execution completion message indicative of completion of execution of the first task routine;
within the first performance container, in response to at least storage of the execution completion message within the task queue, store a job completion message indicative of completion of the job flow within the job queue; and
in response to storage of the job completion message within the job queue, transmit an indication of completion of the job flow to the requesting device via the network.
3. The apparatus of claim 1 , wherein the at least one processor is caused to perform operations comprising:
within the second task container, in response to the storage of the second task routine execution request message within the task queue, perform operations comprising:
use the identifier associated with the second task to retrieve a second task routine of the set of task routines from the at least one federated area; and
commence execution, within the second task container, of instructions of the second task routine to commence performance of the second task.
4. The apparatus of claim 1 , wherein:
the request to perform a job flow comprises a job flow identifier of the job flow definition;
the retrieval of the job flow definition comprises use of the job flow identifier, wherein the job flow definition comprises a set of flow task identifiers that each identify a task of the set of tasks; and
the identifier associated with the first task comprises a flow task identifier of the set of flow task identifiers that identifies the first task.
5. The apparatus of claim 1 , wherein the at least one processor is caused to perform operations comprising:
within a second performance container, execute instructions of a second instance of the performance routine to cause the at least one processor to, in response to the storage of the job performance request message within the job queue and in response to an uninstantiation of the first performance container prior to storage of a job completion message indicative of completion of the job flow within the job queue, perform operations comprising:
derive the order of performance of the set of tasks based on indications in the job flow definitions of dependencies among the set of tasks; and
store, within the task queue, the first task routine execution request message.
6. The apparatus of claim 1 , wherein the at least one processor is caused to perform operations comprising:
within a third task container, in response to the storage of the first task routine execution request message within the task queue and in response to an uninstantiation of the first task container prior to storage, within the task queue, of a task completion message indicative of completion of the first task, perform operations comprising:
use the identifier associated with the first task to retrieve the first task routine from the at least one federated area; and
commence execution, within the third task container, of instructions of the first task routine to commence performance of the first task.
7. The apparatus of claim 1 , wherein:
the at least one processor executes instructions of a resource allocation routine to cause the at least one processor to dynamically allocate multiple pods based on availability of at least one of processing resources and storage resources;
the first performance container and a first messaging container are instantiated within a first pod of the multiple pods;
within the first messaging container, instructions of a first instance of a messaging routine are executed by the at least one processor to cause the at least one processor to provide the first instance of the performance routine with access to the job queue and the task queue;
the first task container and a second messaging container are instantiated within a second pod of the multiple pods; and
within the second messaging container, instructions of a second instance of the messaging routine are executed by the at least one processor to cause the at least one processor to provide the first task routine with access to the task queue.
8. The apparatus of claim 7 , wherein:
a portal container and a third messaging container are instantiated within a third pod of the multiple pods;
within the portal container, the at least one processor executes instructions of an instance of a portal routine to cause the at least one processor to perform operations comprising:
receive the request to perform the job flow from the requesting device;
determine whether the requesting device or an operator of the requesting device is authorized to request a performance of the job flow;
retrieve the job flow definition from the at least one federated area; and
store the job performance request message within the job queue.
9. The apparatus of claim 1 , wherein the at least one processor executes instructions of a message broker routine to provide multiple queues comprising the job queue and the task queue.
10. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause at least one processor to perform operations comprising:
receive, at the at least one processor and from a requesting device via a network, a request to perform a job flow, wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during the job flow performance;
the job flow definition is stored among multiple job flow definitions within at least one federated area;
the set of task routines is stored among multiple task routines within the at least one federated area; and
the at least one federated area is maintained within at least one storage device;
retrieve the job flow definition from among the multiple job flow definitions in the at least one federated area;
store, within a job queue, a job performance request message comprising the job flow definition;
within a first performance container, execute instructions of a first instance of a performance routine to cause the at least one processor to, in response to the storage of the job performance request message within the job queue, perform operations comprising:
derive an order of performance of the set of tasks that specifies at least a first task of the set of tasks to be performed and a second task to be performed after the first task, and that is based on indications in the job flow definition of dependencies among the set of tasks;
store, within a task queue, a first task routine execution request message comprising an identifier associated with the first task; and
in response to at least storage of an execution completion message indicative of completion of execution of the first task routine within the task queue, store, within the task queue, a second task routine execution request message comprising an identifier associated with the second task to cause a latest version of a second task routine to be executed within a second task container to cause the at least one processor to perform the second task; and
within a first task container, in response to the storage of the first task routine execution request message within the task queue, perform operations comprising:
use the identifier associated with the first task to retrieve a latest version of a first task routine of the set of task routines from the at least one federated area;
commence execution, within the first task container, of instructions of the first task routine to commence performance of the first task; and
following completion of the execution of the first task routine, store, within the task queue, the execution completion message indicative of completion of execution of the first task routine.
11. The computer-program product of claim 10 , wherein the at least one processor is caused to perform operations comprising:
within the first task container, following completion of the execution of the instructions of the first task routine, store, within the task queue, an execution completion message indicative of completion of execution of the first task routine;
within the first performance container, in response to at least storage of the execution completion message within the task queue, store a job completion message indicative of completion of the job flow within the job queue; and
in response to storage of the job completion message within the job queue, transmit an indication of completion of the job flow to the requesting device via the network.
12. The computer-program product of claim 10 , wherein the at least one processor is caused to perform operations comprising:
within the second task container, in response to the storage of the second task routine execution request message within the task queue, perform operations comprising:
use the identifier associated with the second task to retrieve a second task routine of the set of task routines from the at least one federated area; and
commence execution, within the second task container, of instructions of the second task routine to commence performance of the second task.
13. The computer-program product of claim 10 , wherein:
the request to perform a job flow comprises a job flow identifier of the job flow definition;
the retrieval of the job flow definition comprises use of the job flow identifier, wherein the job flow definition comprises a set of flow task identifiers that each identify a task of the set of tasks; and
the identifier associated with the first task comprises a flow task identifier of the set of flow task identifiers that identifies the first task the retrieval of the first task routine comprises selecting, from among the multiple task routines stored within the at least one federated area, a latest version of task routine comprising instructions that, when executed by the at least processor, causes the at least one processor to perform the first task.
14. The computer-program product of claim 10 , wherein the at least one processor is caused to perform operations comprising:
within a second performance container, execute instructions of a second instance of the performance routine to cause the at least one processor to, in response to the storage of the job performance request message within the job queue and in response to an uninstantiation of the first performance container prior to storage of a job completion message indicative of completion of the job flow within the job queue, perform operations comprising:
derive the order of performance of the set of tasks based on indications in the job flow definitions of dependencies among the set of tasks; and
store, within the task queue, the first task routine execution request message.
15. The computer-program product of claim 10 , wherein the at least one processor is caused to perform operations comprising:
within a third task container, in response to the storage of the first task routine execution request message within the task queue and in response to an uninstantiation of the first task container prior to storage, within the task queue, of a task completion message indicative of completion of the first task, perform operations comprising:
use the identifier associated with the first task to retrieve the first task routine from the at least one federated area; and
commence execution, within the third task container, of instructions of the first task routine to commence performance of the first task.
16. The computer-program product of claim 10 , wherein:
the at least one processor executes instructions of a resource allocation routine to cause the at least one processor to dynamically allocate multiple pods based on availability of at least one of processing resources and storage resources;
the first performance container and a first messaging container are instantiated within a first pod of the multiple pods;
within the first messaging container, instructions of a first instance of a messaging routine are executed by the at least one processor to cause the at least one processor to provide the first instance of the performance routine with access to the job queue and the task queue;
the first task container and a second messaging container are instantiated within a second pod of the multiple pods; and
within the second messaging container, instructions of a second instance of the messaging routine are executed by the at least one processor to cause the at least one processor to provide the first task routine with access to the task queue.
17. The computer-program product of claim 16 , wherein:
a portal container and a third messaging container are instantiated within a third pod of the multiple pods;
within the portal container, the at least one processor executes instructions of an instance of a portal routine to cause the at least one processor to perform operations comprising:
receive the request to perform the job flow from the requesting device;
determine whether the requesting device or an operator of the requesting device is authorized to request a performance of the job flow;
retrieve the job flow definition from the at least one federated area; and
store the job performance request message within the job queue.
18. The computer-program product of claim 1 , wherein the at least one processor executes instructions of a message broker routine to provide multiple queues comprising the job queue and the task queue.
19. A computer-implemented method comprising:
receiving, by at least one processor and from a requesting device via a network, a request to perform a job flow, wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during the job flow performance;
the job flow definition is stored among multiple job flow definitions within at least one federated area;
the set of task routines is stored among multiple task routines within the at least one federated area; and
the at least one federated area is maintained within at least one storage device;
retrieving the job flow definition from among the multiple job flow definitions in the at least one federated area;
storing, within a job queue, a job performance request message comprising the job flow definition;
within a first performance container, executing, by the at least one processor, instructions of a first instance of a performance routine causing the at least one processor to, in response to the storage of the job performance request message within the job queue, perform operations comprising:
deriving, by the at least one processor, an order of performance of the set of tasks that specifies at least a first task of the set of tasks to be performed and a second task to be performed after the first task, and that is based on indications in the job flow definition of dependencies among the set of tasks;
storing, within a task queue, a first task routine execution request message comprising an identifier associated with the first task; and
in response to at least storage of an execution completion message indicative of completion of execution of the first task routine within the task queue, storing, within the task queue, a second task routine execution request message comprising an identifier associated with the second task to cause a latest version of a second task routine to be executed within a second task container to cause the at least one processor to perform the second task; and
within a first task container, in response to the storage of the first task routine execution request message within the task queue, performing operations comprising:
using the identifier associated with the first task to retrieve a latest version of a first task routine of the set of task routines from the at least one federated area;
commencing execution of, by the at least one processor and within the first task container, instructions of the first task routine to commence performance of the first task; and
following completion of the execution of the first task routine, storing, within the task queue, the execution completion message indicative of completion of execution of the first task routine.
20. The computer-implemented method of claim 19 , comprising:
within the first task container, after completion of the execution of the instructions of the first task routine, storing, within the task queue, an execution completion message indicative of completion of execution of the first task routine;
within the first performance container, in response to at least storage of the execution completion message within the task queue, storing a job completion message indicative of completion of the job flow within the job queue; and
in response to storage of the job completion message within the job queue, transmitting an indication of completion of the job flow to the requesting device via the network.
21. The computer-implemented method of claim 19 , comprising:
within the second task container, in response to the storage of the second task routine execution request message within the task queue, performing operations comprising:
using the identifier associated with the second task to retrieve a second task routine of the set of task routines from the at least one federated area; and
commencing execution of, by the at least one processor and within the second task container, instructions of the second task routine to commence performance of the second task.
22. The computer-implemented method of claim 19 , wherein:
the request to perform a job flow comprises a job flow identifier of the job flow definition;
retrieving the job flow definition comprises using the job flow identifier, wherein the job flow definition comprises a set of flow task identifiers that each identify a task of the set of tasks; and
the identifier associated with the first task comprises a flow task identifier of the set of flow task identifiers that identifies the first task.
23. The computer-implemented method of claim 19 , comprising:
within a second performance container, executing, by the at least one processor, instructions of a second instance of the performance routine causing the at least one processor to, in response to the storage of the job performance request message within the job queue and in response to an uninstantiation of the first performance container prior to storage of a job completion message indicative of completion of the job flow within the job queue, perform operations comprising:
deriving the order of performance of the set of tasks based on indications in the job flow definitions of dependencies among the set of tasks; and
storing, within the task queue, the first task routine execution request message.
24. The computer-implemented method of claim 19 , comprising:
within a third task container, in response to the storage of the first task routine execution request message within the task queue and in response to an uninstantiation of the first task container prior to storage, within the task queue, of a task completion message indicative of completion of the first task, performing operations comprising:
using the identifier associated with the first task to retrieve the first task routine from the at least one federated area; and
commencing execution of, by the at least one processor and within the third task container, instructions of the first task routine to commence performance of the first task.
25. The computer-implemented method of claim 19 , wherein:
the at least one processor executes instructions of a resource allocation routine to cause the at least one processor to dynamically allocate multiple pods based on availability of at least one of processing resources and storage resources;
the first performance container and a first messaging container are instantiated within a first pod of the multiple pods;
within the first messaging container, instructions of a first instance of a messaging routine are executed by the at least one processor causing the at least one processor to provide the first instance of the performance routine with access to the job queue and the task queue;
the first task container and a second messaging container are instantiated within a second pod of the multiple pods; and
within the second messaging container, instructions of a second instance of the messaging routine are executed by the at least one processor causing the at least one processor to provide the first task routine with access to the task queue.
26. The computer-implemented method of claim 25 , wherein:
a portal container and a third messaging container are instantiated within a third pod of the multiple pods;
within the portal container, the at least one processor executes instructions of an instance of a portal routine causing the at least one processor to perform operations comprising:
receiving the request to perform the job flow from the requesting device;
determining whether the requesting device or an operator of the requesting device is authorized to request a performance of the job flow;
retrieving the job flow definition from the at least one federated area; and
storing the job performance request message within the job queue.
27. The computer-implemented method of claim 19 , comprising execute, by the at least one processor, instructions of a message broker routine to provide multiple queues comprising the job queue and the task queue.