Message queue protocol for sequential execution of related task routines in many task computing
An apparatus including a processor to: output a first request message onto a group sub-queue shared by multiple task containers to request execution of a first task routine; within a task container, respond to the first request message, by outputting a first task in-progress message onto an individual sub-queue not shared with other task containers to accede to executing the first task routine, followed by a task completion message; and respond to the task completion message by allowing the task completion message to remain on the individual sub-queue to keep the task container from executing another task routine from another request message on the group sub-queue, outputting a second request message onto the individual sub-queue to cause execution of a second task routine within the same task container to perform a second task, and responding to the second task in-progress message by de-queuing the task completion message.
1. An apparatus comprising at least one processor and a storage to store instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receive, at the at least one processor, and from a requesting device via a network, a request to perform a job flow comprising a set of tasks;
within a performance container, the at least one processor is caused to output a first task execution request message onto a group sub-queue of a task queue to convey, to a set of task containers sharing access to the group sub-queue, a request to execute a first task routine to perform a first task of the set of tasks;
within a first task container of the set of task containers, and in response to the output of the first task execution request message onto the group sub-queue, the at least one processor is caused to perform operations of the first task comprising:
accede to executing the first task routine by outputting a first task in-progress message onto a first individual sub-queue of the task queue, wherein access to the first individual sub-queue is not shared with other task containers;
execute the first task routine to generate at least one portion of a data object as part of performing the first task;
store the at least one portion of the data object within at least one federated area; and
output a first task completion message onto the first individual sub-queue of the task queue; and
within the performance container, and in response to the output of the first task completion message onto the first individual sub-queue, the at least one processor is caused to perform operations comprising:
determine, based on data dependencies among the set of tasks, whether a second task of the set of tasks uses the at least one portion of the data object as an input;
in response to a determination that the second task uses the at least one portion of the data object as an input, perform operations comprising:
while allowing the first task completion message to remain on the first individual sub-queue to cause the first task container to refrain from acceding to executing another task routine from another task routine execution request message on the group sub-queue, output a second task execution request message onto the first individual sub-queue to cause execution of a second task routine within the first task container to perform the second task using a buffered copy of the at least one portion of the data object as input; and
in response to output of a second task in-progress message onto the first individual sub-queue from the first task container to accede to executing the second task routine, de-queue the first task completion message; and
in response to a determination that the second task routine does not use the at least one portion of the data object as input, the at least one processor is caused to de-queue the first task completion message from the first individual sub-queue to enable the first task container to accede to executing another task routine from another task routine execution request message on the group sub-queue.
2. The apparatus of claim 1 , wherein:
each task container of the set of task containers is of a first type that supports executions of multiple instances of task routines at least partially in parallel;
the data object is generated in a distributed form as a set of data object blocks during executions of multiple instances of the first task routine across multiple task containers of the set of task containers, including the execution of the first task routine within the first task container;
the at least one portion of the data object generated during the execution of the first task routine within the first task container comprises a first data object block of the set of data object blocks;
within the performance container, the at least one processor is caused to output a third task execution request message onto the group sub-queue to convey, to the set of task containers, a request to execute the first task routine to perform the first task to generate a second data object block of the set of data object blocks; and
within a second task container of the set of task containers, and in response to the output of the third task execution request message onto the group sub-queue, the at least one processor is caused to perform operations of the first task comprising:
accede to executing the first task routine requested in the third task routine execution request message by outputting a third task in-progress message onto a second individual sub-queue of the task queue, wherein access to the second individual sub-queue is not shared with other task containers; and
execute the first task routine to generate the second data object block as part of performing the first task.
3. The apparatus of claim 1 , wherein:
each task container of the set of task containers is of a first type that supports executions of multiple instances of task routines at least partially in parallel;
within the performance container, the at least one processor is caused to output a third task execution request message onto the group sub-queue to convey, to the set of task containers, a request to execute the first task routine to perform the first task to generate a second data object block of the set of data object blocks;
within the first task container, and in response to the output of the second task execution request message onto the first individual sub-queue, the at least one processor is caused to perform operations of the second task comprising:
accede to executing the second task routine by outputting the second task in-progress message onto the first individual sub-queue of the task queue;
execute the second task routine using the first data object block as an input as part of performing the second task; and
output a second task completion message onto the first individual sub-queue of the task queue;
within the performance container, and in response to the output of the second task completion message onto the first individual sub-queue, the at least one processor is caused to perform operations comprising:
determine, based on the data dependencies among the set of tasks, whether there is another task of the set of tasks that uses data output by the second task as an input; and
in response to a determination that there is not another task that uses data output by the second task as an input, de-queue the second task completion message from the first individual sub-queue to enable the first task container to accede to executing another task routine from another task routine execution request message on the group sub-queue; and
within the first task container, in response to the de-queuing of the second task completion message, and in response to the output of the third task execution request message onto the group sub-queue, the at least one processor is caused to perform further operations of the first task comprising:
accede to executing the first task routine that is requested in the third task routine execution request message by outputting a third task in-progress message onto the first individual sub-queue; and
execute the first task routine to generate the second data object block as part of performing the first task.
4. The apparatus of claim 1 , wherein:
each task container of the set of task containers is of a first type that supports executions of multiple instances of task routines at least partially in parallel;
the at least one processor executes instructions of a resource allocation routine to cause the at least one processor to dynamically allocate multiple containers based on availability of at least one of processing resources and storage resources; and
within the performance container, and in response to commencement of performance of the first task, the at least one processor is caused to provide, to the resource allocation routine, an indication of at least one of a need for provision of more task containers of the first type or a need for provision of fewer task containers of a second type that supports executions of single instances of task routines.
5. The apparatus of claim 4 , wherein:
the task queue is allocated to convey just messages associated with the execution of multiple instances of task routines by the first type of task container; and
another task queue is allocated to convey just messages associated with the execution of single instances of task routines by a second type of task container.
6. The apparatus of claim 1 , wherein:
data objects are stored within the at least one federated area in a format that is associated with syntax of a first programming language in which at least a subset of task routines are written;
the first task container provides a memory space within the first task container to support exchanging a data object generated in a format that is associated with syntax of a second programming language between two task routines written in the second programming language; and
within the first task container, and in response to the first task routine being written in the second programming language, the at least one processor is caused to perform operations comprising:
convert the at least one portion of the first data object into the format associated with the syntax of the first programming language for storage within the at least one federated area, and for being buffered within the device in which the first task container is maintained;
store another copy of the at least one portion of the first data object, as generated by execution of the first task routine in the format associated with the syntax of the second programming language, within the memory space; and
in response to the first task container being caused to execute the second task routine immediately after the execution of the first task routine, and in response to the second task routine also being written in the second programming language, use the copy of the at least one portion of the first data object, as stored in the memory space, as an input to the second task routine.
7. The apparatus of claim 6 , wherein, within the first task container, in response to the first task routine being written in the second programming language, in response to the first task container being caused to execute the second task routine immediately after the execution of the first task routine, and in response to the second task routine being written in the first programming language, the at least one processor is caused to use the buffered copy of the at least one portion of the first data object as an input to the second task routine.
8. The apparatus of claim 1 , wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed by executing a corresponding set of task routines, and that specifies data dependencies among the set of tasks;
the set of tasks comprises the first task and the second task;
the task queue comprises the group sub-queue, and a set of individual sub-queues;
the set of individual sub-queues comprises the first individual sub-queue; and
each individual sub-queue of the set of individual sub-queues is accessible to a different task container of the set of task containers to provide each task container of the set of task containers with a path of communication with the performance container that is not shared with any other task container.
9. The apparatus of claim 1 , wherein:
the group sub-queue is maintained throughout at least the performance of the job flow;
the first individual sub-queue is newly instantiated each time the first task container accedes to executing a task routine that is requested in a task routine execution request message that is output onto the group sub-queue;
acceding to executing the first task routine comprises instantiating the first individual sub-queue before outputting the first task in-progress message onto the first individual sub-queue; and
in response to the determination that the second task does not use the at least one portion of the first data object as an input, and in response to de-queuing of the first task completion message from the first individual sub-queue, the at least one processor is caused to uninstantiate the first individual sub-queue.
10. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause at least one processor to perform operations comprising:
receive, at the at least one processor, and from a requesting device via a network, a request to perform a job flow comprising a set of tasks;
within a performance container, the at least one processor is caused to output a first task execution request message onto a group sub-queue of a task queue to convey, to a set of task containers sharing access to the group sub-queue, a request to execute a first task routine to perform a first task of the set of tasks;
within a first task container of the set of task containers, and in response to the output of the first task execution request message onto the group sub-queue, the at least one processor is caused to perform operations of the first task comprising:
accede to executing the first task routine by outputting a first task in-progress message onto a first individual sub-queue of the task queue, wherein access to the first individual sub-queue is not shared with other task containers;
execute the first task routine to generate at least one portion of a data object as part of performing the first task;
store the at least one portion of the data object within at least one federated area; and
output a first task completion message onto the first individual sub-queue of the task queue; and
within the performance container, and in response to the output of the first task completion message onto the first individual sub-queue, the at least one processor is caused to perform operations comprising:
determine, based on data dependencies among the set of tasks, whether a second task of the set of tasks uses the at least one portion of the data object as an input;
in response to a determination that the second task uses the at least one portion of the data object as an input, perform operations comprising:
while allowing the first task completion message to remain on the first individual sub-queue to cause the first task container to refrain from acceding to executing another task routine from another task routine execution request message on the group sub-queue, output a second task execution request message onto the first individual sub-queue to cause execution of a second task routine within the first task container to perform the second task using a buffered copy of the at least one portion of the data object as input; and
in response to output of a second task in-progress message onto the first individual sub-queue from the first task container to accede to executing the second task routine, de-queue the first task completion message; and
in response to a determination that the second task routine does not use the at least one portion of the data object as input, the at least one processor is caused to de-queue the first task completion message from the first individual sub-queue to enable the first task container to accede to executing another task routine from another task routine execution request message on the group sub-queue.
11. The computer-program product of claim 10 , wherein:
each task container of the set of task containers is of a first type that supports executions of multiple instances of task routines at least partially in parallel;
the data object is generated in a distributed form as a set of data object blocks during executions of multiple instances of the first task routine across multiple task containers of the set of task containers, including the execution of the first task routine within the first task container;
the at least one portion of the data object generated during the execution of the first task routine within the first task container comprises a first data object block of the set of data object blocks;
within the performance container, the at least one processor is caused to output a third task execution request message onto the group sub-queue to convey, to the set of task containers, a request to execute the first task routine to perform the first task to generate a second data object block of the set of data object blocks; and
within a second task container of the set of task containers, and in response to the output of the third task execution request message onto the group sub-queue, the at least one processor is caused to perform operations of the first task comprising:
accede to executing the first task routine requested in the third task routine execution request message by outputting a third task in-progress message onto a second individual sub-queue of the task queue, wherein access to the second individual sub-queue is not shared with other task containers; and
execute the first task routine to generate the second data object block as part of performing the first task.
12. The computer-program product of claim 10 , wherein:
each task container of the set of task containers is of a first type that supports executions of multiple instances of task routines at least partially in parallel;
within the performance container, the at least one processor is caused to output a third task execution request message onto the group sub-queue to convey, to the set of task containers, a request to execute the first task routine to perform the first task to generate a second data object block of the set of data object blocks;
within the first task container, and in response to the output of the second task execution request message onto the first individual sub-queue, the at least one processor is caused to perform operations of the second task comprising:
accede to executing the second task routine by outputting the second task in-progress message onto the first individual sub-queue of the task queue;
execute the second task routine using the first data object block as an input as part of performing the second task; and
output a second task completion message onto the first individual sub-queue of the task queue;
within the performance container, and in response to the output of the second task completion message onto the first individual sub-queue, the at least one processor is caused to perform operations comprising:
determine, based on the data dependencies among the set of tasks, whether there is another task of the set of tasks that uses data output by the second task as an input; and
in response to a determination that there is not another task that uses data output by the second task as an input, de-queue the second task completion message from the first individual sub-queue to enable the first task container to accede to executing another task routine from another task routine execution request message on the group sub-queue; and
within the first task container, in response to the de-queuing of the second task completion message, and in response to the output of the third task execution request message onto the group sub-queue, the at least one processor is caused to perform further operations of the first task comprising:
accede to executing the first task routine that is requested in the third task routine execution request message by outputting a third task in-progress message onto the first individual sub-queue; and
execute the first task routine to generate the second data object block as part of performing the first task.
13. The computer-program product of claim 10 , wherein:
each task container of the set of task containers is of a first type that supports executions of multiple instances of task routines at least partially in parallel;
the at least one processor executes instructions of a resource allocation routine to cause the at least one processor to dynamically allocate multiple containers based on availability of at least one of processing resources and storage resources; and
within the performance container, and in response to commencement of performance of the first task, the at least one processor is caused to provide, to the resource allocation routine, an indication of at least one of a need for provision of more task containers of the first type or a need for provision of fewer task containers of a second type that supports executions of single instances of task routines.
14. The computer-program product of claim 13 , wherein:
the task queue is allocated to convey just messages associated with the execution of multiple instances of task routines by the first type of task container; and
another task queue is allocated to convey just messages associated with the execution of single instances of task routines by a second type of task container.
15. The computer-program product of claim 10 , wherein:
data objects are stored within the at least one federated area in a format that is associated with syntax of a first programming language in which at least a subset of task routines are written;
the first task container provides a memory space within the first task container to support exchanging a data object generated in a format that is associated with syntax of a second programming language between two task routines written in the second programming language; and
within the first task container, and in response to the first task routine being written in the second programming language, the at least one processor is caused to perform operations comprising:
convert the at least one portion of the first data object into the format associated with the syntax of the first programming language for storage within the at least one federated area, and for being buffered within the device in which the first task container is maintained;
store another copy of the at least one portion of the first data object, as generated by execution of the first task routine in the format associated with the syntax of the second programming language, within the memory space; and
in response to the first task container being caused to execute the second task routine immediately after the execution of the first task routine, and in response to the second task routine also being written in the second programming language, use the copy of the at least one portion of the first data object, as stored in the memory space, as an input to the second task routine.
16. The computer-program product of claim 15 , wherein, within the first task container, in response to the first task routine being written in the second programming language, in response to the first task container being caused to execute the second task routine immediately after the execution of the first task routine, and in response to the second task routine being written in the first programming language, the at least one processor is caused to use the buffered copy of the at least one portion of the first data object as an input to the second task routine.
17. The computer-program product of claim 10 , wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed by executing a corresponding set of task routines, and that specifies data dependencies among the set of tasks;
the set of tasks comprises the first task and the second task;
the task queue comprises the group sub-queue, and a set of individual sub-queues;
the set of individual sub-queues comprises the first individual sub-queue; and
each individual sub-queue of the set of individual sub-queues is accessible to a different task container of the set of task containers to provide each task container of the set of task containers with a path of communication with the performance container that is not shared with any other task container.
18. The computer-program product of claim 10 , wherein:
the group sub-queue is maintained throughout at least the performance of the job flow;
the first individual sub-queue is newly instantiated each time the first task container accedes to executing a task routine that is requested in a task routine execution request message that is output onto the group sub-queue;
acceding to executing the first task routine comprises instantiating the first individual sub-queue before outputting the first task in-progress message onto the first individual sub-queue; and
in response to the determination that the second task does not use the at least one portion of the first data object as an input, and in response to de-queuing of the first task completion message from the first individual sub-queue, the at least one processor is caused to uninstantiate the first individual sub-queue.
19. A computer-implemented method comprising:
receiving, at the at least one processor, and from a requesting device via a network, a request to perform a job flow comprising a set of tasks;
within a performance container, outputting a first task execution request message onto a group sub-queue of a task queue to convey, to a set of task containers sharing access to the group sub-queue, a request to execute a first task routine to perform a first task of the set of tasks;
within a first task container of the set of task containers, and in response to the output of the first task execution request message onto the group sub-queue, performing operations of the first task comprising:
acceding to executing the first task routine by outputting a first task in-progress message onto a first individual sub-queue of the task queue, wherein access to the first individual sub-queue is not shared with other task containers;
executing, by the at least one processor, the first task routine to generate at least one portion of a data object as part of performing the first task;
storing the at least one portion of the data object within at least one federated area; and
outputting a first task completion message onto the first individual sub-queue of the task queue; and
within the performance container, and in response to the output of the first task completion message onto the first individual sub-queue, performing operations comprising:
determining, by the at least one processor, and based on data dependencies among the set of tasks, whether a second task of the set of tasks uses the at least one portion of the data object as an input;
in response to a determination, by the at least one processor, that the second task uses the at least one portion of the data object as an input, performing operations comprising:
while allowing the first task completion message to remain on the first individual sub-queue to cause the first task container to refrain from acceding to executing another task routine from another task routine execution request message on the group sub-queue, outputting a second task execution request message onto the first individual sub-queue to cause execution of a second task routine within the first task container to perform the second task using a buffered copy of the at least one portion of the data object as input; and
in response to output of a second task in-progress message onto the first individual sub-queue from the first task container to accede to executing the second task routine, de-queuing the first task completion message; and
in response to a determination, by the at least one processor, that the second task routine does not use the at least one portion of the data object as input, de-queuing the first task completion message from the first individual sub-queue to enable the first task container to accede to executing another task routine from another task routine execution request message on the group sub-queue.
20. The computer-implemented method of claim 19 , wherein:
each task container of the set of task containers is of a first type that supports executions of multiple instances of task routines at least partially in parallel;
the data object is generated in a distributed form as a set of data object blocks during executions of multiple instances of the first task routine across multiple task containers of the set of task containers, including the execution of the first task routine within the first task container;
the at least one portion of the data object generated during the execution of the first task routine within the first task container comprises a first data object block of the set of data object blocks; and
the method further comprises:
within the performance container, outputting a third task execution request message onto the group sub-queue to convey, to the set of task containers, a request to execute the first task routine to perform the first task to generate a second data object block of the set of data object blocks; and
within a second task container of the set of task containers, and in response to the output of the third task execution request message onto the group sub-queue, performing operations of the first task comprising:
acceding to executing the first task routine requested in the third task routine execution request message by outputting a third task in-progress message onto a second individual sub-queue of the task queue, wherein access to the second individual sub-queue is not shared with other task containers; and
executing, by the at least one processor, the first task routine to generate the second data object block as part of performing the first task.
21. The computer-implemented method of claim 19 , wherein:
each task container of the set of task containers is of a first type that supports executions of multiple instances of task routines at least partially in parallel; and
the method further comprises:
within the performance container, outputting a third task execution request message onto the group sub-queue to convey, to the set of task containers, a request to execute the first task routine to perform the first task to generate a second data object block of the set of data object blocks;
within the first task container, and in response to the output of the second task execution request message onto the first individual sub-queue, performing operations of the second task comprising:
acceding to executing the second task routine by outputting the second task in-progress message onto the first individual sub-queue of the task queue;
executing, by the at least one processor, the second task routine using the first data object block as an input as part of performing the second task; and
outputting a second task completion message onto the first individual sub-queue of the task queue;
within the performance container, and in response to the output of the second task completion message onto the first individual sub-queue, performing operations comprising:
determining, by the at least one processor, and based on the data dependencies among the set of tasks, whether there is another task of the set of tasks that uses data output by the second task as an input; and
in response to a determination, by the at least one processor, that there is not another task that uses data output by the second task as an input, de-queuing the second task completion message from the first individual sub-queue to enable the first task container to accede to executing another task routine from another task routine execution request message on the group sub-queue; and
within the first task container, in response to the de-queuing of the second task completion message, and in response to the output of the third task execution request message onto the group sub-queue, performing further operations of the first task comprising:
acceding to executing the first task routine that is requested in the third task routine execution request message by outputting a third task in-progress message onto the first individual sub-queue; and
executing, by the at least one processor, the first task routine to generate the second data object block as part of performing the first task.
22. The computer-implemented method of claim 19 , wherein:
each task container of the set of task containers is of a first type that supports executions of multiple instances of task routines at least partially in parallel;
the at least one processor executes instructions of a resource allocation routine to cause the at least one processor to dynamically allocate multiple containers based on availability of at least one of processing resources and storage resources; and
the method further comprises, within the performance container, and in response to commencement of performance of the first task, providing, to the resource allocation routine, an indication of at least one of a need for provision of more task containers of the first type or a need for provision of fewer task containers of a second type that supports executions of single instances of task routines.
23. The computer-implemented method of claim 22 , wherein:
the task queue is allocated to convey just messages associated with the execution of multiple instances of task routines by the first type of task container; and
another task queue is allocated to convey just messages associated with the execution of single instances of task routines by a second type of task container.
24. The computer-implemented method of claim 19 , wherein:
data objects are stored within the at least one federated area in a format that is associated with syntax of a first programming language in which at least a subset of task routines are written;
the first task container provides a memory space within the first task container to support exchanging a data object generated in a format that is associated with syntax of a second programming language between two task routines written in the second programming language; and
the method further comprises, within the first task container, and in response to the first task routine being written in the second programming language, performing operations comprising:
converting, by the at least one processor, the at least one portion of the first data object into the format associated with the syntax of the first programming language for storage within the at least one federated area, and for being buffered within the device in which the first task container is maintained;
storing another copy of the at least one portion of the first data object, as generated by execution of the first task routine in the format associated with the syntax of the second programming language, within the memory space; and
in response to the first task container being caused to execute the second task routine immediately after the execution of the first task routine, and in response to the second task routine also being written in the second programming language, using, by the at least one processor the copy of the at least one portion of the first data object, as stored in the memory space, as an input to the second task routine.
25. The computer-implemented method of claim 24 , comprising, within the first task container, in response to the first task routine being written in the second programming language, in response to the first task container being caused to execute the second task routine immediately after the execution of the first task routine, and in response to the second task routine being written in the first programming language, using the buffered copy of the at least one portion of the first data object as an input to the second task routine.
26. The computer-implemented method of claim 19 , wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed by executing a corresponding set of task routines, and that specifies data dependencies among the set of tasks;
the set of tasks comprises the first task and the second task;
the task queue comprises the group sub-queue, and a set of individual sub-queues;
the set of individual sub-queues comprises the first individual sub-queue; and
each individual sub-queue of the set of individual sub-queues is accessible to a different task container of the set of task containers to provide each task container of the set of task containers with a path of communication with the performance container that is not shared with any other task container.
27. The computer-implemented method of claim 19 , wherein:
the group sub-queue is maintained throughout at least the performance of the job flow;
the first individual sub-queue is newly instantiated each time the first task container accedes to executing a task routine that is requested in a task routine execution request message that is output onto the group sub-queue;
acceding to executing the first task routine comprises instantiating the first individual sub-queue before outputting the first task in-progress message onto the first individual sub-queue; and
the method further comprises, in response to the determination that the second task does not use the at least one portion of the first data object as an input, and in response to de-queuing of the first task completion message from the first individual sub-queue, uninstantiating the first individual sub-queue.