Annotation of a machine learning pipeline with operational semantics to support distributed lineage tracking
A system, computer program product, and method are provided for distributed data workflow semantics. A pipeline, such as a machine learning (ML) pipeline, is represented in a data flow graph (DFG). The represented pipeline is subject to annotations, with the annotations including pipeline nodes and object references. The pre-processed pipeline is subject to execution or processing with the annotated object references capturing object lineage. Output from the executed pipeline is constructed and a corresponding control signal is formatted to dynamically and selectively control an operatively coupled physical hardware device or software.
1. A computer system comprising:
a processor set:
one or more computer readable storage media; and
program instructions stored on the one or more computer readable storage media to cause the processor set to perform operations comprising:
pre-processing a pipeline configured to train a machine learning (ML) model, the pipeline represented in a data flow graph (DFG), including:
annotating one or more nodes of the DFG with two or more operational semantics for pipeline operations; and
selectively annotating one or more output object references to a corresponding input object and a prior node state for an output object from the DFG;
executing the pipeline represented in the DFG with the selectively annotated one or more output object references, including capturing an object lineage to provide data of how an object in the pipeline is produced with respect to the output object using the selectively annotated one or more output object references;
identifying provenance of one or more objects represented in the pipeline corresponding to a generated output including performance of the executed pipeline using the object lineage;
selectively applying a remediation action to the DFG based on the provenance of the one or more objects corresponding to the generated output; and
restarting the pipeline by executing the pipeline from a location in a sub-graph of the DFG where the remediation action was selectively applied.
2. The computer system of claim 1 , wherein the operations further comprise:
dynamically configuring and issuing a control signal configuration, the control signal configuration based on a constructed output to an operatively coupled physical hardware device, a process controlled by software, or a combination thereof, associated with the DFG, the control signal configuration selectively controls a physical state of the operatively coupled physical hardware device, the software, or a combination thereof.
3. The computer system of claim 2 , wherein the executing of the pipeline represented in the DFG with the one or more output object references comprises:
tracking transformation of data across the pipeline.
4. The computer system of claim 3 , wherein the identifying of the provenance of the one or more objects comprises:
identifying the sub-graph in the DFG and an input object reference that produced the constructed output.
5. The computer system of claim 1 , wherein capturing of the object lineage comprises:
retaining a posterior node state for the one or more output object references from the DFG.
6. The computer system of claim 5 , wherein the operations further comprise:
attaching the object lineage as object metadata.
7. A computer program product comprising:
one or more computer readable storage media; and
program instructions stored on the one or more computer readable storage media to perform operations comprising:
pre-processing a pipeline represented in a data flow graph (DFG), the pipeline configured to train a machine learning (ML) model, including:
annotating one or more nodes of the DFG with two or more operational semantics for pipeline operations; and
selectively annotating one or more output object references to a corresponding input object and a prior node state for an output object from the DFG;
executing the pipeline represented in the DFG with the selectively annotated one or more output object references, including capturing an object lineage to provide data of how an object in the pipeline is produced with respect to the output object using the selectively annotated one or more output object references;
identifying provenance of one or more objects represented in the pipeline corresponding to a generated output including performance of the executed pipeline using the object lineage;
selectively applying a remediation action to the DFG based on the provenance of the one or more objects corresponding to the generated output; and
restarting the pipeline by executing the pipeline from a location in a sub-graph of the DFG where the remediation action was selectively applied.
8. The computer program product of claim 7 , wherein the operations further comprise:
dynamically configuring and issuing a control signal configuration, the control signal configuration based on a constructed output to an operatively coupled physical hardware device, a process controlled by software, or a combination thereof, associated with the DFG, the control signal configuration selectively controls a physical state of the operatively coupled physical hardware device, the software, or a combination thereof.
9. The computer program product of claim 8 , wherein the executing the pipeline represented in the DFG with the one or more output object references comprises:
tracking transformation of data across the pipeline.
10. The computer program product of claim 9 , wherein the identifying the provenance of the one or more objects comprises:
identifying the sub-graph in the DFG and an input object reference that produced the constructed output.
11. The computer program product of claim 7 , wherein the capturing the object lineage comprises:
retaining a posterior node state for the one or more output object references from the DFG.
12. The computer program product of claim 11 , wherein the operations further comprise:
attaching the object lineage as object metadata.
13. A computer implemented method comprising:
pre-processing a pipeline represented in a data flow graph (DFG), the pipeline configured to train a machine learning (ML) model, including:
annotating one or more nodes of the DFG with two or more operational semantics for pipeline operations; and
selectively annotating one or more output object references to a corresponding input object and a prior node state for an output object from the DFG;
executing the pipeline represented in the DFG with the selectively annotated one or more output object references, including capturing an object lineage to provide data of how an object in the pipeline is produced with respect to the output object using the selectively annotated one or more output object references;
identifying provenance of one or more objects represented in the pipeline corresponding to a generated output including performance of the executed pipeline using the object lineage;
selectively applying a remediation action to the DFG based on the provenance of the one or more objects corresponding to the generated output; and
restarting the pipeline by executing the pipeline from a location in a sub-graph of the DFG where the remediation action was selectively applied.
14. The computer implemented method of claim 13 , further comprising:
dynamically configuring and issuing a control signal configuration, the control signal configuration based on a constructed output to an operatively coupled physical hardware device, a process controlled by software, or a combination thereof, associated with the DFG, the control signal configuration selectively controls a physical state of the operatively coupled physical hardware device, the software, or a combination thereof.
15. The computer implemented method of claim 14 , wherein executing the pipeline represented in the DFG with the one or more output object references further comprises:
tracking transformation of data across the pipeline.
16. The computer implemented method of claim 15 , wherein identifying the provenance of the one or more objects further comprises:
identifying the sub-graph in the DFG and an input object reference that produced the constructed output.
17. The computer implemented method of claim 13 , wherein capturing the object lineage further comprises:
retaining a posterior node state for the one or more output object references from the DFG.
18. The computer implemented method of claim 17 , further comprising:
attaching the object lineage as object metadata.
19. The computer implemented method of claim 13 , wherein the two or more operational semantics comprises an input combination, a firing combination, and a state, or a combination thereof.