Annotation of a machine learning pipeline with operational semantics
A system, computer program product, and method are provided for distributed data workflow semantics. A pipeline, such as a machine learning pipeline, is represented in a data flow graph (DFG) with nodes and edges. The represented nodes are configured to be annotated with an operational semantic. On order of execution of the pipeline is discovered through the node annotation(s) represented in the annotated DFG, and execution of the pipeline is based on the discovered order. A control signal formatted based on the executed pipeline is configured to dynamically and selectively control an operatively coupled physical hardware device.
1. A computer system comprising:
a processor operatively coupled to memory;
an artificial intelligence (AI) platform, operatively coupled to the processor, comprising:
a pipeline manager configured to represent a pipeline in a data flow graph (DFG) with individual nodes representing an instance of a mathematical operation and individual edges representing an object;
a processing manager configured to pre-processing the pipeline represented in the DFG, including selectively annotate one or more of the nodes in the DFG with two or more operational semantics, the semantics comprising an input combination, a firing combination, a state of the node, an output condition, or a combination thereof; and
an evaluator configured to evaluate the annotated DFG and control a scheduling order responsive to the evaluation, including discover an order of execution based on the node annotations; and
a director configured to execute the pipeline represented in the annotated DFG responsive to the evaluation, and construct output from the executed pipeline, the constructed output configured to align with an output condition semantic.
2. The computer system of claim 1 , further comprising the director configured to dynamically configure and issue a control signal, the control signal configuration based on the constructed output to an operatively coupled physical hardware device, a process controlled by software, or a combination thereof, associated with the annotated DFG, the control signal to selectively control a physical state of the operatively coupled device, the software, or a combination thereof.
3. The computer system of claim 2 , wherein execution of the pipeline represented in the annotated DFG responsive to the evaluation includes creating a representation of the object in the annotated DFG.
4. The computer system of claim 3 , wherein the object is a pointer to a potentially unrealized object.
5. The computer system of claim 2 , wherein evaluation of the annotated DFG and control of the scheduling order includes the evaluator to determine node input and node state.
6. The computer system of claim 1 , further comprising the processing manager configured to concurrently submit two or more tasks across two or more nodes in the annotated DFG.
7. The computer system of claim 1 , wherein the pipeline is a machine learning pipeline.
8. A computer program product configured to interface with a computer readable storage medium having program code embodied therewith, the program code executable by a processor to:
represent a pipeline in a data flow graph (DFG) with individual nodes representing an instance of a mathematical operation and individual edges representing an object;
pre-process the pipeline represented in the DFG, the pre-processing including program code configured to selectively annotate one or more of the nodes with two or more operational semantics, the semantics comprising an input combination, a firing combination, a state of the node, an output condition, or a combination thereof;
evaluate the annotated DFG and control a scheduling order responsive to the evaluation, including discover an order of execution based on the node annotations; and
execute the pipeline represented in the annotated DFG responsive to the evaluation, and construct output from the executed pipeline, the constructed output configured to align with an output condition semantic.
9. The computer program product of claim 8 , further comprising program code configured to dynamically configure and issue a control signal, the control signal configuration based on the constructed output to an operatively coupled physical hardware device, a process controlled by software, or a combination thereof, associated with the annotated DFG, the control signal configured to selectively control a physical state of the operatively coupled device, the software, or a combination thereof.
10. The computer program product of claim 9 , wherein executing the pipeline represented in the annotated DFG responsive to the evaluation includes creating representation of the object in the annotated DEG.
11. The computer program product of claim 10 , wherein the object is a pointer to a potentially unrealized object.
12. The computer program product of claim 9 , wherein evaluation of the annotated DFG and control of the scheduling order includes program code to determine node input and node state.
13. The computer program product of claim 8 , further comprising program code configured to concurrently submit two or more tasks across two or more nodes in the annotated DFG.
14. The computer program product of claim 8 , wherein the pipeline is a machine learning pipeline.
15. A computer implemented method comprising:
representing a pipeline in a data flow graph (DFG) with individual nodes representing an instance of a mathematical operation and individual edges representing an object;
pre-processing the pipeline represented in the DFG, the pre-processing including selectively annotating one or more of the nodes with two or more operational semantics, the semantics comprising an input combination, a firing combination, a state of the node, an output condition, or a combination thereof;
evaluating the annotated DFG and controlling a scheduling order responsive to the evaluation, including discovering an order of execution based on the node annotations; and
executing the pipeline represented in the annotated DFG responsive to the evaluation, and constructing output from the executed pipeline, the constructed output configured to align with an output condition semantic.
16. The computer implemented method of claim 15 , further comprising dynamically configuring and issuing a control signal, the control signal configuration based on the constructed output to an operatively coupled physical hardware device, a process controlled by software, or a combination thereof, associated with the annotated DFG, the control signal configured to selectively control a physical state of the operatively coupled device, the software, or a combination thereof.
17. The computer implemented method of claim 16 ,
wherein executing the pipeline represented in the annotated DFG responsive to the evaluation includes creating a representation of the object in the annotated DEG.
18. The computer implemented method of claim 17 , wherein the object is a pointer to a potentially unrealized object.
19. The computer implemented method of claim 16 , wherein evaluating the annotated DFG and controlling the scheduling order includes determining node input and node state.
20. The computer system of claim 1 , wherein the output semantics are directed to flattening the output object.