Workflow engine tool
A workflow engine tool is disclosed that enables scientists and engineers to programmatically author workflows (e.g., a directed acyclic graph, “DAG”) with nearly no overhead, using a simpler script that needs almost no modifications for portability among multiple different workflow engines. This permits users to focus on the business logic of the project, avoiding the distracting tedious overhead related to workflow management (such as uploading modules, drawing edges, setting parameters, and other tasks). The workflow engine tool provides an abstraction layer on top of workflow engines, introducing a binding function that converts a programming language function (e.g., a normal python function) into a workflow module definition. The workflow engine tool infers module instances and induces edge dependencies automatically by inferring from a programming language script to build a DAG.
1. A workflow engine tool comprising:
a processor; and
a computer storage medium storing instructions that are operative when executed by the processor to:
provide a programming language script with at least one function;
provide, for the at least one function, a function definition including an input, a received parameter, and an output, wherein the input of the at least one function dictates execution flow dependency between nodes and is data generated dynamically by an upstream node while running a first target workflow engine, wherein the received parameter of the at least one function is received prior to running the first target workflow engine, and wherein the upstream node is a unit of task execution in an upstream workflow;
identify the first target workflow engine;
convert the at least one function into a workflow module definition to generate a first workflow pipeline for the first target workflow engine; and
run the first target workflow engine to execute the first workflow pipeline.
2. The workflow engine tool of claim 1 wherein the instructions are further operative to:
control a workflow interface component to send the converted workflow module definition to the first target workflow engine; and
execute an underlying system call for the first target workflow engine.
3. The workflow engine tool of claim 1 wherein the instructions are further operative to:
present the programming language script in a graphical user interface (GUI) to a user.
4. The workflow engine tool of claim 1 wherein the instructions are further operative to:
identify a second target workflow engine; and
generate a second workflow pipeline for the second target workflow engine.
5. The workflow engine tool of claim 1 wherein:
the received parameter of the at least one function includes an enforced data type comprising at least one of an integer, a floating point number, a string, or a date-time.
6. The workflow engine tool of claim 1 wherein a programming language of the programming language script includes a python programming language, and wherein the first workflow pipeline comprises a directed acyclic graph (DAG).
7. The workflow engine tool of claim 1 wherein the instructions are further operative to:
extract module definitions from the programming language script;
extract execution flow information from the programming language script;
generate, for the first target workflow engine, modules from the extracted module definitions;
generate, for the first target workflow engine, edges for connecting the modules generated for the first target workflow engine; and
connect the modules generated for the first target workflow engine with the edges generated for the first target workflow engine, based at least on the extracted execution flow information.
8. A method of workflow engine management, the method comprising:
providing a programming language script with at least one function;
providing, for the at least one function, a function definition including an input, a received parameter, and an output, wherein the input of the at least one function dictates execution flow dependency between nodes and is data generated dynamically by an upstream node while running a first target workflow engine, wherein the received parameter of the at least one function is received prior to running the first target workflow engine, and wherein the upstream node is a unit of task execution in an upstream workflow;
identifying the first target workflow engine;
converting the at least one function into a workflow module definition to generate a first workflow pipeline for the first target workflow engine; and
running the first target workflow engine to execute the first workflow pipeline.
9. The method of claim 8 further comprising:
controlling a workflow interface component to send the converted workflow module definition to the first target workflow engine; and
executing an underlying system call for the first target workflow engine.
10. The method of claim 8 further comprising:
presenting the programming language script in a graphical user interface (GUI) to a user.
11. The method of claim 8 further comprising:
identifying a second target workflow engine; and
generating a second workflow pipeline for the second target workflow engine.
12. The method of claim 8 wherein:
the received parameter of the at least one function includes an enforced data type comprising at least one of an integer, a floating point number, a string, or a date-time.
13. The method of claim 8 wherein a programming language of the programming language script includes a python programming language, and wherein the first workflow pipeline comprises a directed acyclic graph (DAG).
14. The method of claim 8 wherein converting the at least one function into the workflow module definition comprises:
extracting module definitions from the programming language script;
extracting execution flow information from the programming language script;
generating, for the first target workflow engine, modules from the extracted module definitions;
generating, for the first target workflow engine, edges for connecting the modules generated for the first target workflow engine; and
connecting the modules generated for the first target workflow engine with the edges generated for the first target workflow engine, based at least on the extracted execution flow information.
15. One or more computer storage media having computer-executable instructions stored thereon for workflow engine management, which, on execution by a computer, cause the computer to perform operations comprising:
providing a programming language script with at least one function;
providing, for the at least one function, a function definition including an input, a received parameter, and an output, wherein the input of the at least one function dictates execution flow dependency between nodes and is data generated dynamically by an upstream node while running a first target workflow engine, wherein the received parameter of the at least one function is received prior to running the first target workflow engine, and wherein the upstream node is a unit of task execution in an upstream workflow;
identifying the first target workflow engine;
converting the at least one function into a workflow module definition to generate a first workflow pipeline for the first target workflow engine; and
running the first target workflow engine to execute the first workflow pipeline.
16. The one or more computer storage media of claim 15 wherein the operations further comprise:
controlling a workflow interface component to send the converted workflow module definition to the first target workflow engine; and
executing an underlying system call for the first target workflow engine.
17. The one or more computer storage media of claim 15 wherein the operations further comprise:
presenting the programming language script in a graphical user interface (GUI) to a user.
18. The one or more computer storage media of claim 15 wherein the operations further comprise:
identifying a second target workflow engine; and
generating a second workflow pipeline for the second target workflow engine.
19. The one or more computer storage media of claim 15 wherein:
the received parameter of the at least one function includes an enforced data type comprising at least one of an integer, a floating point number, a string, or a date-time.
20. The one or more computer storage media of claim 15 wherein a programming language of the programming language script includes a python programming language, and wherein the first workflow pipeline comprises a directed acyclic graph (DAG).