Implicit status in many task computing
An apparatus includes a processor to: within a performance container, execute a performance routine to derive an order of performance of tasks of a job flow based on dependencies, begin performing the tasks, and store, within a job queue, a job performance status indication including task performance statuses; identify a set of sub flows within the job flow based on branches in the job flow; correlate each of the task performance statuses to a corresponding sub flow performance status; reduce the job performance status indication size by, for each sub flow in which all tasks have been completed, replace the corresponding task performance statuses with the corresponding sub flow performance status of completed, and for each sub flow with no task performed, replace the corresponding task performance statuses with the corresponding sub flow performance status of not executed; and transmit the job performance status indication to the requesting device.
1. An apparatus comprising at least one processor and a storage to store instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
receive, at the at least one processor and from a requesting device via a network, a first request for status of a requested performance of a job flow, wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during a performance of the job flow, and that specifies data dependencies among the set of tasks;
store, within a job queue, a job performance status request message comprising a job flow instance identifier assigned to the requested performance;
within a performance container, execute instructions of an instance of a performance routine to cause the at least one processor to perform operations comprising:
derive an order of performance of the set of tasks based on indications in the job flow definition of dependencies among the set of tasks;
store, within a task queue, multiple task routine execution request messages in an order that is based on the derived order of performance and on corresponding task completion messages stored within the task queue, to cause at least a subset of the set of task routines to be executed in the derived order of performance to cause at least a corresponding subset of the set of tasks to be performed in the derived order of performance to perform the job flow, resulting in the storage of the corresponding task completion messages; and
in response to the storage of the job performance status request message within the job queue, store, within the job queue, a job performance status indication comprising a set of task performance statuses for the set of tasks that are based on the task completion messages stored within the task queue;
analyze indications in the job flow definition of dependencies among the set of tasks to identify a set of sub flows within the job flow based on branches in the job flow, wherein each sub flow comprises multiple tasks of the set of tasks that are to be performed sequentially;
based on which tasks of the set of tasks are included within each sub flow, correlate each of the task performance statuses in the job performance status indication to a corresponding sub flow performance status;
reduce a size of the job performance status indication from the size at which the job performance status indication is stored in the job queue by performing operations comprising:
for each sub flow in which all tasks of the sub flow have been completed, replace the corresponding task performance statuses with the corresponding sub flow performance status of completed; and
for each sub flow in which no task has been performed and in which no task is currently being performed, replace the corresponding task performance statuses with the corresponding sub flow performance status of not yet executed; and
following the reduction in size of the job performance status indication, transmit the job performance status indication to the requesting device.
2. The apparatus of claim 1 , wherein the at least one processor is caused to perform operations comprising:
receive, at the at least one processor and from the requesting device via the network, a request to perform the job flow;
generate, within a database, an entry for the requested performance of the job flow, wherein the entry comprises:
a job flow instance identifier assigned to the requested performance of the job flow;
a set of task instance identifiers that are each assigned to a task of the set of tasks; and
an overall indication of status of the requested performance of the job flow;
store, within the job queue, a job flow performance request message comprising:
the job flow definition;
the job flow instance identifier; and
the set of task instance identifiers;
within the performance container, in response to the storage of the job flow performance request message within the job queue, store, within the job queue, a job in progress message comprising the job flow instance identifier and an identifier of the instance of the performance routine to indicate commencement of the requested performance of the job flow by the instance of the performance routine; and
in response to the storage of the job in progress message within the job queue, store an indication of a running status as the overall indication of status of the requested performance of the job flow.
3. The apparatus of claim 2 , wherein the at least one processor is caused to perform operations comprising:
receive, at the at least one processor and from the requesting device via the network, a second request for status of the requested performance of the job flow prior to storage of the job in progress message within the job queue; and
in response to receiving the second request for status, store an indication of a not executed status as the overall indication of status of the requested performance of the job flow.
4. The apparatus of claim 2 , wherein within the performance container, the at least one processor is caused to perform operations comprising:
prior to causing the performance of at least the corresponding subset of the set of tasks, generate the job performance status indication to indicate a task performance status of not executed for each task of the set of tasks;
include store, within a task routine execution request message of the multiple task routine execution request messages, the job flow instance identifier and a task instance identifier of the set of task instance identifiers that is associated with a task of the set of tasks; and
in response to storage of a task in progress message in the task queue that comprises the job flow instance identifier and the task instance identifier to indicate commencement of the performance of the task, store a task performance status of running for the task within the job performance status indication.
5. The apparatus of claim 4 , wherein within the performance container, the at least one processor is caused to perform operations comprising:
in response to storage of a task completion message in the task queue that comprises the job flow instance identifier and the task instance identifier to indicate completion of the performance of the task, store a task performance status of completed for the task within the job performance status indication.
6. The apparatus of claim 4 , wherein within the performance container, the at least one processor is caused, in response to storage of a task cancelation message in the task queue that comprises the job flow instance identifier and the task instance identifier to indicate cancelation of the performance of the task, to perform operations comprising:
store a task performance status of canceled for the task within the job performance status indication; and
store a job cancelation message in the job queue that comprises the job flow instance identifier to indicate cancelation of the requested performance of the job flow.
7. The apparatus of claim 6 , wherein, in response to the storage of the job cancelation message in the job queue, the at least one processor is caused to perform operations comprising:
store an indication of a canceled status as the overall indication of status of the requested performance of the job flow; and
transmit an indication of cancelation of the requested performance of the job flow to the requesting device.
8. The apparatus of claim 1 , wherein:
within at least one sub flow of the set of sub flows, a task was being performed when the job performance status indication was stored within the job queue;
the job performance status indication includes a corresponding task performance status of running for the task; and
the at least one processor is caused to reduce the size of the job performance status indication by performing operations comprising:
for the sub flow that includes the task, replace the task performance statuses corresponding to the multiple tasks within the sub flow with a corresponding sub flow performance status of running, and an identifier of the task.
9. The apparatus of claim 1 , wherein:
during the requested performance of the job flow, an attempt to perform a task of the set of tasks failed;
the job performance status indication includes a corresponding task performance status of failed for the task; and
the at least one processor is caused to reduce the size of the job performance status indication by performing operations comprising:
for the sub flow that includes the task, replace the task performance statuses corresponding to the multiple tasks within the sub flow with a corresponding sub flow performance status of failed, and an identifier of the task.
10. The apparatus of claim 1 , wherein:
during the requested performance of the job flow, a request is received from the requesting device to cancel the requested performance of the job flow;
in response to the request to cancel, a performance of a task of the set of tasks is canceled after commencing and before being completed;
the job performance status indication includes a corresponding task performance status of canceled for the task; and
the at least one processor is caused to reduce the size of the job performance status indication by performing operations comprising:
for the sub flow that includes the task, replace the task performance statuses corresponding to the multiple tasks within the sub flow with a corresponding sub flow performance status of canceled, and an identifier of the task.
11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause at least one processor to perform operations comprising:
receive, at the at least one processor and from a requesting device via a network, a first request for status of a requested performance of a job flow, wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during a performance of the job flow, and that specifies data dependencies among the set of tasks;
store, within a job queue, a job performance status request message comprising a job flow instance identifier assigned to the requested performance;
within a performance container, execute instructions of an instance of a performance routine to cause the at least one processor to perform operations comprising:
derive an order of performance of the set of tasks based on indications in the job flow definition of dependencies among the set of tasks;
store, within a task queue, multiple task routine execution request messages in an order that is based on the derived order of performance and on corresponding task completion messages stored within the task queue, to cause at least a subset of the set of task routines to be executed in the derived order of performance to cause at least a corresponding subset of the set of tasks to be performed in the derived order of performance to perform the job flow, resulting in the storage of the corresponding task completion messages; and
in response to the storage of the job performance status request message within the job queue, store, within the job queue, a job performance status indication comprising a set of task performance statuses for the set of tasks that are based on the task completion messages stored within the task queue;
analyze indications in the job flow definition of dependencies among the set of tasks to identify a set of sub flows within the job flow based on branches in the job flow, wherein each sub flow comprises multiple tasks of the set of tasks that are to be performed sequentially;
based on which tasks of the set of tasks are included within each sub flow, correlate each of the task performance statuses in the job performance status indication to a corresponding sub flow performance status;
reduce a size of the job performance status indication from the size at which the job performance status indication is stored in the job queue by performing operations comprising:
for each sub flow in which all tasks of the sub flow have been completed, replace the corresponding task performance statuses with the corresponding sub flow performance status of completed; and
for each sub flow in which no task has been performed and in which no task is currently being performed, replace the corresponding task performance statuses with the corresponding sub flow performance status of not yet executed; and
following the reduction in size of the job performance status indication, transmit the job performance status indication to the requesting device.
12. The computer-program product of claim 11 , wherein the at least one processor is caused to perform operations comprising:
receive, at the at least one processor and from the requesting device via the network, a request to perform the job flow;
generate, within a database, an entry for the requested performance of the job flow, wherein the entry comprises:
a job flow instance identifier assigned to the requested performance of the job flow;
a set of task instance identifiers that are each assigned to a task of the set of tasks; and
an overall indication of status of the requested performance of the job flow;
store, within the job queue, a job flow performance request message comprising:
the job flow definition;
the job flow instance identifier; and
the set of task instance identifiers;
within the performance container, in response to the storage of the job flow performance request message within the job queue, store, within the job queue, a job in progress message comprising the job flow instance identifier and an identifier of the instance of the performance routine to indicate commencement of the requested performance of the job flow by the instance of the performance routine; and
in response to the storage of the job in progress message within the job queue, store an indication of a running status as the overall indication of status of the requested performance of the job flow.
13. The computer-program product of claim 12 , wherein the at least one processor is caused to perform operations comprising:
receive, at the at least one processor and from the requesting device via the network, a second request for status of the requested performance of the job flow prior to storage of the job in progress message within the job queue; and
in response to receiving the second request for status, store an indication of a not executed status as the overall indication of status of the requested performance of the job flow.
14. The computer-program product of claim 12 , wherein within the performance container, the at least one processor is caused to perform operations comprising:
prior to causing the performance of at least the corresponding subset of the set of tasks, generate the job performance status indication to indicate a task performance status of not executed for each task of the set of tasks;
include, within a task routine execution request message of the multiple task routine execution request messages, the job flow instance identifier and a task instance identifier of the set of task instance identifiers that is associated with a task of the set of tasks; and
in response to storage of a task in progress message in the task queue that comprises the job flow instance identifier and the task instance identifier to indicate commencement of the performance of the task, store a task performance status of running for the task within the job performance status indication.
15. The computer-program product of claim 14 , wherein within the performance container, the at least one processor is caused to perform operations comprising:
in response to storage of a task completion message in the task queue that comprises the job flow instance identifier and the task instance identifier to indicate completion of the performance of the task, store a task performance status of completed for the task within the job performance status indication.
16. The computer-program product of claim 14 , wherein within the performance container, the at least one processor is caused, in response to storage of a task cancelation message in the task queue that comprises the job flow instance identifier and the task instance identifier to indicate cancelation of the performance of the task, to perform operations comprising:
store a task performance status of canceled for the task within the job performance status indication; and
store a job cancelation message in the job queue that comprises the job flow instance identifier to indicate cancelation of the requested performance of the job flow.
17. The computer-program product of claim 16 , wherein, in response to the storage of the job cancelation message in the job queue, the at least one processor is caused to perform operations comprising:
store an indication of a canceled status as the overall indication of status of the requested performance of the job flow; and
transmit an indication of cancelation of the requested performance of the job flow to the requesting device.
18. The computer-program product of claim 11 , wherein:
within at least one sub flow of the set of sub flows, a task was being performed when the job performance status indication was stored within the job queue;
the job performance status indication includes a corresponding task performance status of running for the task; and
the at least one processor is caused to reduce the size of the job performance status indication by performing operations comprising:
for the sub flow that includes the task, replace the task performance statuses corresponding to the multiple tasks within the sub flow with a corresponding sub flow performance status of running, and an identifier of the task.
19. The computer-program product of claim 11 , wherein:
during the requested performance of the job flow, an attempt to perform a task of the set of tasks failed;
the job performance status indication includes a corresponding task performance status of failed for the task; and
the at least one processor is caused to reduce the size of the job performance status indication by performing operations comprising:
for the sub flow that includes the task, replace the task performance statuses corresponding to the multiple tasks within the sub flow with a corresponding sub flow performance status of failed, and an identifier of the task.
20. The computer-program product of claim 11 , wherein:
during the requested performance of the job flow, a request is received from the requesting device to cancel the requested performance of the job flow;
in response to the request to cancel, a performance of a task of the set of tasks is canceled after commencing and before being completed;
the job performance status indication includes a corresponding task performance status of canceled for the task; and
the at least one processor is caused to reduce the size of the job performance status indication by performing operations comprising:
for the sub flow that includes the task, replace the task performance statuses corresponding to the multiple tasks within the sub flow with a corresponding sub flow performance status of canceled, and an identifier of the task.
21. A computer-implemented method comprising:
receiving, by at least one processor and from a requesting device via a network, a first request for status of a requested performance of a job flow, wherein:
the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during a performance of the job flow, and that specifies data dependencies among the set of tasks;
storing, within a job queue, a job performance status request message comprising a job flow instance identifier assigned to the requested performance;
within a performance container, executing, by the at least one processor, instructions of an instance of a performance routine to cause the at least one processor to perform operations comprising:
deriving, by the at least one processor, an order of performance of the set of tasks based on indications in the job flow definition of dependencies among the set of tasks;
storing, within a task queue, multiple task routine execution request messages in an order that is based on the derived order of performance and on corresponding task completion messages stored within the task queue, to cause at least a subset of the set of task routines to be executed in the derived order of performance to cause, by the at least one processor, at least a corresponding subset of the set of tasks to be performed in the derived order of performance to perform the job flow, resulting in the storage of the corresponding task completion messages; and
in response to the storage of the job performance status request message within the job queue, storing, within the job queue, a job performance status indication comprising a set of task performance statuses for the set of tasks that are based on the task completion messages stored within the task queue;
analyzing, by the at least one processor, indications in the job flow definition of dependencies among the set of tasks to identify a set of sub flows within the job flow based on branches in the job flow, wherein each sub flow comprises multiple tasks of the set of tasks that are to be performed sequentially;
based on which tasks of the set of tasks are included within each sub flow, correlating, by the at least one processor, each of the task performance statuses in the job performance status indication to a corresponding sub flow performance status;
reducing, by the at least one processor, a size of the job performance status indication from the size at which the job performance status indication is stored in the job queue by performing operations comprising:
for each sub flow in which all tasks of the sub flow have been completed, replacing the corresponding task performance statuses with the corresponding sub flow performance status of completed; and
for each sub flow in which no task has been performed and in which no task is currently being performed, replacing the corresponding task performance statuses with the corresponding sub flow performance status of not yet executed; and
following the reduction in size of the job performance status indication, transmitting, from the at least one processor, the job performance status indication to the requesting device.
22. The computer-implemented method of claim 21 , comprising:
receiving, by the at least one processor and from the requesting device via the network, a request to perform the job flow;
generating, by the at least one processor, within a database, an entry for the requested performance of the job flow, wherein the entry comprises:
a job flow instance identifier assigned to the requested performance of the job flow;
a set of task instance identifiers that are each assigned to a task of the set of tasks; and
an overall indication of status of the requested performance of the job flow;
storing, within the job queue, a job flow performance request message comprising:
the job flow definition;
the job flow instance identifier; and
the set of task instance identifiers;
within the performance container, in response to the storage of the job flow performance request message within the job queue, storing, within the job queue, a job in progress message comprising the job flow instance identifier and an identifier of the instance of the performance routine to indicate commencement of the requested performance of the job flow by the instance of the performance routine; and
in response to the storage of the job in progress message within the job queue, storing an indication of a running status as the overall indication of status of the requested performance of the job flow.
23. The computer-implemented method of claim 22 , comprising:
receiving, by the at least one processor and from the requesting device via the network, a second request for status of the requested performance of the job flow prior to storage of the job in progress message within the job queue; and
in response to receiving the second request for status, storing an indication of a not executed status as the overall indication of status of the requested performance of the job flow.
24. The computer-implemented method of claim 22 , comprising, within the performance container, performing operations comprising:
prior to causing the performance of at least the corresponding subset of the set of tasks, generating, by the at least one processor, the job performance status indication to indicate a task performance status of not executed for each task of the set of tasks;
including, within a task routine execution request message of the multiple task routine execution request messages, the job flow instance identifier and a task instance identifier of the set of task instance identifiers that is associated with a task of the set of tasks; and
in response to storage of a task in progress message in the task queue that comprises the job flow instance identifier and the task instance identifier to indicate commencement of the performance of the task, storing a task performance status of running for the task within the job performance status indication.
25. The computer-implemented method of claim 24 , comprising, within the performance container, performing operations comprising:
in response to storage of a task completion message in the task queue that comprises the job flow instance identifier and the task instance identifier to indicate completion of the performance of the task, storing a task performance status of completed for the task within the job performance status indication.
26. The computer-implemented method of claim 24 , comprising, within the performance container, in response to storage of a task cancelation message in the task queue that comprises the job flow instance identifier and the task instance identifier to indicate cancelation of the performance of the task, performing operations comprising:
storing a task performance status of canceled for the task within the job performance status indication; and
storing a job cancelation message in the job queue that comprises the job flow instance identifier to indicate cancelation of the requested performance of the job flow.
27. The computer-implemented method of claim 26 , comprising, in response to the storage of the job cancelation message in the job queue, performing operations comprising:
storing an indication of a canceled status as the overall indication of status of the requested performance of the job flow; and
transmitting an indication of cancelation of the requested performance of the job flow to the requesting device.
28. The computer-implemented method of claim 21 , wherein:
within at least one sub flow of the set of sub flows, a task was being performed when the job performance status indication was stored within the job queue;
the job performance status indication includes a corresponding task performance status of running for the task; and
the method comprises reducing the size of the job performance status indication by performing operations comprising:
for the sub flow that includes the task, replacing the task performance statuses corresponding to the multiple tasks within the sub flow with a corresponding sub flow performance status of running, and an identifier of the task.
29. The computer-implemented method of claim 21 , wherein:
during the requested performance of the job flow, an attempt to perform a task of the set of tasks failed;
the job performance status indication includes a corresponding task performance status of failed for the task; and
the method comprises reducing the size of the job performance status indication by performing operations comprising:
for the sub flow that includes the task, replacing the task performance statuses corresponding to the multiple tasks within the sub flow with a corresponding sub flow performance status of failed, and an identifier of the task.
30. The computer-implemented method of claim 21 , wherein:
during the requested performance of the job flow, a request is received from the requesting device to cancel the requested performance of the job flow;
in response to the request to cancel, a performance of a task of the set of tasks is canceled after commencing and before being completed;
the job performance status indication includes a corresponding task performance status of canceled for the task; and
the method comprises reducing the size of the job performance status indication by performing operations comprising:
for the sub flow that includes the task, replacing the task performance statuses corresponding to the multiple tasks within the sub flow with a corresponding sub flow performance status of canceled, and an identifier of the task.