Automated generation of job flow definitions
An apparatus includes a processor to: receive a request to generate a superset job flow replacing multiple job flows including an output job flow and preceding job flows previously performed to generate an output data object; identify a first subset of mid-flow data object(s) generated by preceding job flow(s) as input(s) to the output job flow to generate the output data object; identify a second subset of the mid-flow data object(s) generated by preceding job flow(s) as input(s) to other preceding job flow(s) generating the first subset; in response to a lack of a second subset, derive the superset job flow and/or corresponding DAG to include at least one task of the output job flow and at least one task of each preceding job flow that generated the first subset; and transmit an indication of the generation of the superset job flow.
1. An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:
receive, at the processor, and from a remote device via a network, a request to generate a superset job flow that incorporates multiple job flows, wherein:
each job flow of the multiple job flows is defined by a corresponding job flow definition as a set of tasks to be performed by execution of a corresponding set of task routines;
each performance of a job flow of the multiple job flows is documented by a corresponding instance log that specifies at least one data object used as an input to the job flow and at least one other data object generated as an output of the job flow during the performance;
the multiple job flows comprise an output job flow that has been performed to generate an output data object that the superset job flow is to be capable of re-generating;
the multiple job flows comprise at least one preceding job flow that has been performed to generate at least one mid-flow data object necessary to the performance of the output job flow to generate the output data object; and
for each job flow of the multiple job flows, the job flow definition, the corresponding set of tasks routines, and at least one instance log that documents at least one performance of the job flow are stored within at least one federated area; and
analyze the instance log that documents the performance of the output job flow that generated the output data object to identify the job flow definition of the output job flow and to identify a first subset of the at least one mid-flow data object, wherein each mid-flow data object of the first subset was generated as an output by a preceding job flow and used as input to the output job flow;
use the identification of each mid-flow data object of the first subset to identify at least one instance log that documents a performance of a preceding job flow that generated a mid-flow data object of the first subset;
analyze each instance log of the at least one instance log to identify a job flow definition of the corresponding preceding job flow and to identify a second subset of the at least one mid-flow data object, wherein each mid-flow data object of the second subset was generated as an output by a preceding job flow and used as an input to another preceding job flow;
in response to a lack of identification of a mid-flow data object in the second subset, generate a superset job flow definition that defines the superset job flow as a set of tasks that comprises at least one task specified in the job flow definition of the output job flow and at least one task specified in the job flow definition of each preceding job flow that generated a mid-flow data object of the first subset;
generate a directed acyclic graph (DAG) of the superset job flow based on the superset job flow definition;
store the DAG in the at least one federated area; and
transmit an indication of the generation of the superset job flow to the remote device.
2. The apparatus of claim 1 , wherein in response to an identification of a mid-flow data object in the second subset, the processor is caused to generate the a superset job flow definition that defines the superset job flow as a set of tasks that comprises at least one task specified in the job flow definition of the output job flow, at least one task specified in the job flow definition of each preceding job flow that generated a mid-flow data object of the first subset, and at least one task specified in the job flow definition of each preceding job flow that generated a mid-flow data object of the second subset.
3. The apparatus of claim 1 , wherein the processor is caused to:
analyze the job flow definition of the output job flow to identify at least one task of the output job flow that was not necessary to the generation of the output data object;
prune the at least one task from the output job flow to generate a pruned form of the output job flow;
limit the identification of the first subset of the at least one mid-flow data object to one or more mid-flow data objects that were used as an input to the pruned form of the output job flow; and
limit the at least one task specified in the job flow definition of the output job flow that is included in the superset job flow definition to at least one task remaining in the pruned form of the output job flow.
4. The apparatus of claim 3 , wherein the processor is caused, for each preceding job flow that generated a mid-flow data object of the first subset, to perform operations comprising:
analyze the job flow definition of the preceding job flow to identify at least one task of the preceding job flow that was not necessary to the generation of a mid-flow data object of the first subset;
prune the at least one task from the preceding job flow to generate a pruned form of the preceding job flow;
limit the identification of the second subset of the at least one mid-flow data object to one or more mid-flow data objects that were used as an input to the pruned form of the preceding job flow; and
limit the at least one task specified in the job flow definition of each preceding job flow that is included in the superset job flow definition to at least one task remaining in the pruned form of the preceding job flow.
5. The apparatus of claim 1 , wherein:
the request comprises a data object identifier that identifies the output data object; and
the processor is caused to perform operations comprising:
use the data object identifier of the output data object to search, within the one or more federated areas, for the instance log that documents the performance of the output job flow that generated the output data object; and
use, from the instance log that documents the performance of the output job flow that generated the output data object of the output job flow, a job flow identifier that identifies the output job flow to search for the job flow definition of the output job flow within the one or more federated areas.
6. The apparatus of claim 1 , wherein:
the request comprises a job flow identifier of the output job flow; and
the processor is caused to perform operations comprising:
use the job flow identifier of the output job flow to search, within the one or more federated areas, for the job flow definition for the output job flow;
analyze the job flow definition of the output job flow to determine a quantity of data objects that are output by the output job flow;
use the job flow identifier of the output job flow to search, within the one or more federated areas, for any instance log that documents a performance of the output job flow; and
in response to an identification of no more than one instance log that documents a performance of the output job flow, and in response to a determination that the output job flow outputs more than one data object, transmit a prompt to the remote device to provide an indication of a selection of the output data object from among the more than one data objects.
7. The apparatus of claim 6 , wherein, in response to identification of more than one instance log that documents a performance of the output job flow, the processor is caused to transmit a prompt to the remote device to provide an indication of at least one of a selection of a performance from among the performances of the output job flow documented by the more than one instance log and a selection of the output data object.
8. The apparatus of claim 1 , wherein the processor is caused to perform operations comprising:
generate a series of DAGs that provide a visual indication of each preceding job flow in relation to the output job flow, wherein the series of DAGs includes the DAG of the superset job flow; and
transmit the series of DAGs to the remote device to enable a visual presentation of the series of DAGs by the remote device.
9. The apparatus of claim 1 , wherein:
the remote device or personnel operating the remote device have been granted access to a subset of storage space provided for the storage of objects by the one or more DAGs;
the search for a job flow definition for each preceding job flow is limited to the subset of storage space; and
in response to identification of an instance log that documents a performance of a preceding job flow that generated a mid-flow data object of the first subset or of the second subset, and in response to lack of finding the job flow definition of the preceding job flow within the subset of storage space, the processor is caused to perform operations comprising:
generate the DAG of the superset job flow to include a visual indication of an inability to include the preceding job flow in the superset job flow arising from the inability to find the job flow definition for the preceding job flow; and
transmit the DAG of the superset job flow to the remote device to enable a visual presentation of the DAG with the visual indication.
10. The apparatus of claim 1 , wherein:
the remote device or personnel operating the remote device have been granted access to a subset of storage space provided for the storage of objects by the one or more DAGs;
the search for objects necessary to repeat each performance of each preceding job flow is limited to the subset of storage space; and
in response to identification of an instance log that documents a performance of a preceding job flow that generated a mid-flow data object of the first subset or of the second subset, and in response to a lack of finding a data object used as an input to the performance of the preceding job flow within the subset of storage space, the processor is caused to perform operations comprising:
generate the DAG of the superset job flow to include a visual indication of an inability to include the preceding job flow in the superset job flow arising from the inability to find the data object used as an input to the performance of the preceding job flow; and
transmit the DAG of the superset job flow to the remote device to enable a visual presentation of the DAG with the visual indication.
11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:
receive, at the processor, and from a remote device via a network, a request to generate a superset job flow that incorporates multiple job flows, wherein:
each job flow of the multiple job flows is defined by a corresponding job flow definition as a set of tasks to be performed by execution of a corresponding set of task routines;
each performance of a job flow of the multiple job flows is documented by a corresponding instance log that specifies at least one data object used as an input to the job flow and at least one other data object generated as an output of the job flow during the performance;
the multiple job flows comprise an output job flow that has been performed to generate an output data object that the superset job flow is to be capable of re-generating;
the multiple job flows comprise at least one preceding job flow that has been performed to generate at least one mid-flow data object necessary to the performance of the output job flow to generate the output data object; and
for each job flow of the multiple job flows, the job flow definition, the corresponding set of tasks routines, and at least one instance log that documents at least one performance of the job flow are stored within at least one federated area; and
analyze the instance log that documents the performance of the output job flow that generated the output data object to identify the job flow definition of the output job flow and to identify a first subset of the at least one mid-flow data object, wherein each mid-flow data object of the first subset was generated as an output by a preceding job flow and used as input to the output job flow;
use the identification of each mid-flow data object of the first subset to identify at least one instance log that documents a performance of a preceding job flow that generated a mid-flow data object of the first subset;
analyze each instance log of the at least one instance log to identify a job flow definition of the corresponding preceding job flow and to identify a second subset of the at least one mid-flow data object, wherein each mid-flow data object of the second subset was generated as an output by a preceding job flow and used as an input to another preceding job flow;
in response to a lack of identification of a mid-flow data object in the second subset, generate a superset job flow definition that defines the superset job flow as a set of tasks that comprises at least one task specified in the job flow definition of the output job flow and at least one task specified in the job flow definition of each preceding job flow that generated a mid-flow data object of the first subset;
generate a directed acyclic graph (DAG) of the superset job flow based on the superset job flow definition;
store the DAG in the at least one federated area; and
transmit an indication of the generation of the superset job flow to the remote device.
12. The computer-program product of claim 11 , wherein in response to an identification of a mid-flow data object in the second subset, the processor is caused to generate the a superset job flow definition that defines the superset job flow as a set of tasks that comprises at least one task specified in the job flow definition of the output job flow, at least one task specified in the job flow definition of each preceding job flow that generated a mid-flow data object of the first subset, and at least one task specified in the job flow definition of each preceding job flow that generated a mid-flow data object of the second subset.
13. The computer-program product of claim 11 , wherein the processor is caused:
analyze the job flow definition of the output job flow to identify at least one task of the output job flow that was not necessary to the generation of the output data object;
prune the at least one task from the output job flow to generate a pruned form of the output job flow;
limit the identification of the first subset of the at least one mid-flow data object to one or more mid-flow data objects that were used as an input to the pruned form of the output job flow; and
limit the at least one task specified in the job flow definition of the output job flow that is included in the superset job flow definition to at least one task remaining in the pruned form of the output job flow.
14. The computer-program product of claim 13 , wherein the processor is caused, for each preceding job flow that generated a mid-flow data object of the first subset, to perform operations comprising:
analyze the job flow definition of the preceding job flow to identify at least one task of the preceding job flow that was not necessary to the generation of a mid-flow data object of the first subset;
prune the at least one task from the preceding job flow to generate a pruned form of the preceding job flow;
limit the identification of the second subset of the at least one mid-flow data object to one or more mid-flow data objects that were used as an input to the pruned form of the preceding job flow; and
limit the at least one task specified in the job flow definition of each preceding job flow that is included in the superset job flow definition to at least one task remaining in the pruned form of the preceding job flow.
15. The computer-program product of claim 11 , wherein:
the request comprises a data object identifier that identifies the output data object; and
the processor is caused to perform operations comprising:
use the data object identifier of the output data object to search, within the one or more federated areas, for the instance log that documents the performance of the output job flow that generated the output data object; and
use, from the instance log that documents the performance of the output job flow that generated the output data object of the output job flow, a job flow identifier that identifies the output job flow to search for the job flow definition of the output job flow within the one or more federated areas.
16. The computer-program product of claim 11 , wherein:
the request comprises a job flow identifier of the output job flow; and
the processor is caused to perform operations comprising:
use the job flow identifier of the output job flow to search, within the one or more federated areas, for the job flow definition for the output job flow;
analyze the job flow definition of the output job flow to determine a quantity of data objects that are output by the output job flow;
use the job flow identifier of the output job flow to search, within the one or more federated areas, for any instance log that documents a performance of the output job flow; and
in response to an identification of no more than one instance log that documents a performance of the output job flow, and in response to a determination that the output job flow outputs more than one data object, transmit a prompt to the remote device to provide an indication of a selection of the output data object from among the more than one data objects.
17. The computer-program product of claim 16 , wherein, in response to identification of more than one instance log that documents a performance of the output job flow, the processor is caused to transmit a prompt to the remote device to provide an indication of at least one of a selection of a performance from among the performances of the output job flow documented by the more than one instance log and a selection of the output data object.
18. The computer-program product of claim 11 , wherein the processor is caused to perform operations comprising:
generate a series of DAGs that provide a visual indication of each preceding job flow in relation to the output job flow, wherein the series of DAGs includes the DAG of the superset job flow; and
transmit the series of DAGs to the remote device to enable a visual presentation of the series of DAGs by the remote device.
19. The computer-program product of claim 11 , wherein:
the remote device or personnel operating the remote device have been granted access to a subset of storage space provided for the storage of objects by the one or more DAGs;
the search for a job flow definition for each preceding job flow is limited to the subset of storage space; and
in response to identification of an instance log that documents a performance of a preceding job flow that generated a mid-flow data object of the first subset or of the second subset, and in response to lack of finding the job flow definition of the preceding job flow within the subset of storage space, the processor is caused to perform operations comprising:
generate the DAG of the superset job flow to include a visual indication of an inability to include the preceding job flow in the superset job flow arising from the inability to find the job flow definition for the preceding job flow; and
transmit the DAG of the superset job flow to the remote device to enable a visual presentation of the DAG with the visual indication.
20. The computer-program product of claim 11 , wherein:
the remote device or personnel operating the remote device have been granted access to a subset of storage space provided for the storage of objects by the one or more DAGs;
the search for objects necessary to repeat each performance of each preceding job flow is limited to the subset of storage space; and
in response to identification of an instance log that documents a performance of a preceding job flow that generated a mid-flow data object of the first subset or of the second subset, and in response to a lack of finding a data object used as an input to the performance of the preceding job flow within the subset of storage space, the processor is caused to perform operations comprising:
generate the DAG of the superset job flow to include a visual indication of an inability to include the preceding job flow in the superset job flow arising from the inability to find the data object used as an input to the performance of the preceding job flow; and
transmit the DAG of the superset job flow to the remote device to enable a visual presentation of the DAG with the visual indication.
21. A computer-implemented method comprising:
receiving, by a processor, and from a remote device via a network, a request to generate a superset job flow that incorporates multiple job flows, wherein:
each job flow of the multiple job flows is defined by a corresponding job flow definition as a set of tasks to be performed by execution of a corresponding set of task routines;
each performance of a job flow of the multiple job flows is documented by a corresponding instance log that specifies at least one data object used as an input to the job flow and at least one other data object generated as an output of the job flow during the performance;
the multiple job flows comprise an output job flow that has been performed to generate an output data object that the superset job flow is to be capable of re-generating;
the multiple job flows comprise at least one preceding job flow that has been performed to generate at least one mid-flow data object necessary to the performance of the output job flow to generate the output data object; and
for each job flow of the multiple job flows, the job flow definition, the corresponding set of tasks routines, and at least one instance log that documents at least one performance of the job flow are stored within at least one federated area; and
analyzing, by the processor, the instance log that documents the performance of the output job flow that generated the output data object to identify the job flow definition of the output job flow and to identify a first subset of the at least one mid-flow data object, wherein each mid-flow data object of the first subset was generated as an output by a preceding job flow and used as input to the output job flow;
using, by the processor, the identification of each mid-flow data object of the first subset to identify at least one instance log that documents a performance of a preceding job flow that generated a mid-flow data object of the first subset;
analyzing, by the processor, each instance log of the at least one instance log to identify a job flow definition of the corresponding preceding job flow and to identify a second subset of the at least one mid-flow data object, wherein each mid-flow data object of the second subset was generated as an output by a preceding job flow and used as an input to another preceding job flow;
in response to a lack of identification of a mid-flow data object in the second subset, generating, by the processor, a superset job flow definition that defines the superset job flow as a set of tasks that comprises at least one task specified in the job flow definition of the output job flow and at least one task specified in the job flow definition of each preceding job flow that generated a mid-flow data object of the first subset;
generating, by the processor, a directed acyclic graph (DAG) of the superset job flow based on the superset job flow definition;
storing the DAG in the at least one federated area; and
transmitting, from the processor, an indication of the generation of the superset job flow to the remote device.
22. The computer-implemented method of claim 21 , comprising, in response to an identification of a mid-flow data object in the second subset, generating, by the processor, the a superset job flow definition that defines the superset job flow as a set of tasks that comprises at least one task specified in the job flow definition of the output job flow, at least one task specified in the job flow definition of each preceding job flow that generated a mid-flow data object of the first subset, and at least one task specified in the job flow definition of each preceding job flow that generated a mid-flow data object of the second subset.
23. The computer-implemented method of claim 21 , comprising:
analyzing, by the processor, the job flow definition of the output job flow to identify at least one task of the output job flow that was not necessary to the generation of the output data object;
pruning, by the processor, the at least one task from the output job flow to generate a pruned form of the output job flow;
limiting, by the processor, the identification of the first subset of the at least one mid-flow data object to one or more mid-flow data objects that were used as an input to the pruned form of the output job flow; and
limiting, by the processor, the at least one task specified in the job flow definition of the output job flow that is included in the superset job flow definition to at least one task remaining in the pruned form of the output job flow.
24. The computer-implemented method of claim 23 , comprising, for each preceding job flow that generated a mid-flow data object of the first subset, performing operations comprising:
analyzing, by the processor, the job flow definition of the preceding job flow to identify at least one task of the preceding job flow that was not necessary to the generation of a mid-flow data object of the first subset;
pruning, by the processor, the at least one task from the preceding job flow to generate a pruned form of the preceding job flow;
limiting, by the processor, the identification of the second subset of the at least one mid-flow data object to one or more mid-flow data objects that were used as an input to the pruned form of the preceding job flow; and
limiting, by the processor, the at least one task specified in the job flow definition of each preceding job flow that is included in the superset job flow definition to at least one task remaining in the pruned form of the preceding job flow.
25. The computer-implemented method of claim 21 , wherein:
the request comprises a data object identifier that identifies the output data object; and
the method comprises:
using, by the processor, the data object identifier of the output data object to search, within the one or more federated areas, for the instance log that documents the performance of the output job flow that generated the output data object; and
using, by the processor and from the instance log that documents the performance of the output job flow that generated the output data object of the output job flow, a job flow identifier that identifies the output job flow to search for the job flow definition of the output job flow within the one or more federated areas.
26. The computer-implemented method of claim 21 , wherein:
the request comprises a job flow identifier of the output job flow; and
the method comprises:
using, by the processor, the job flow identifier of the output job flow to search, within the one or more federated areas, for the job flow definition for the output job flow;
analyzing, by the processor, the job flow definition of the output job flow to determine a quantity of data objects that are output by the output job flow;
using, by the processor, the job flow identifier of the output job flow to search, within the one or more federated areas, for any instance log that documents a performance of the output job flow; and
in response to an identification of no more than one instance log that documents a performance of the output job flow, and in response to a determination that the output job flow outputs more than one data object, transmitting, from the processor, a prompt to the remote device to provide an indication of a selection of the output data object from among the more than one data objects.
27. The computer-implemented method of claim 26 , comprising, in response to identification of more than one instance log that documents a performance of the output job flow, transmitting, from the processor, a prompt to the remote device to provide an indication of at least one of a selection of a performance from among the performances of the output job flow documented by the more than one instance log and a selection of the output data object.
28. The computer-implemented method of claim 21 , comprising:
generating, by the processor, a series of DAGs that provide a visual indication of each preceding job flow in relation to the output job flow, wherein the series of DAGs includes the DAG of the superset job flow; and
transmitting, from the processor, the series of DAGs to the remote device to enable a visual presentation of the series of DAGs by the remote device.
29. The computer-implemented method of claim 21 , wherein:
the remote device or personnel operating the remote device have been granted access to a subset of storage space provided for the storage of objects by the one or more DAGs;
the search for a job flow definition for each preceding job flow is limited to the subset of storage space; and
the method comprises, in response to identification of an instance log that documents a performance of a preceding job flow that generated a mid-flow data object of the first subset or of the second subset, and in response to lack of finding the job flow definition of the preceding job flow within the subset of storage space, performing operations comprising:
generating, by the processor, the DAG of the superset job flow to include a visual indication of an inability to include the preceding job flow in the superset job flow arising from the inability to find the job flow definition for the preceding job flow; and
transmitting, by the processor, the DAG of the superset job flow to the remote device to enable a visual presentation of the DAG with the visual indication.
30. The computer-implemented method of claim 21 , wherein:
the remote device or personnel operating the remote device have been granted access to a subset of storage space provided for the storage of objects by the one or more DAGs;
the search for objects necessary to repeat each performance of each preceding job flow is limited to the subset of storage space; and
the method comprises, in response to identification of an instance log that documents a performance of a preceding job flow that generated a mid-flow data object of the first subset or of the second subset, and in response to a lack of finding a data object used as an input to the performance of the preceding job flow within the subset of storage space, performing operations comprising:
generate the DAG of the superset job flow to include a visual indication of an inability to include the preceding job flow in the superset job flow arising from the inability to find the data object used as an input to the performance of the preceding job flow; and
transmit the DAG of the superset job flow to the remote device to enable a visual presentation of the DAG with the visual indication.