Systems and methods for processing computational workflows
In one embodiment, a method of processing a computational workflow comprises receiving a description of a computational workflow. The description comprises a plurality of steps, in which each step has at least one input and at least one output, and further wherein an input from a second step depends on an output from a first step. The description is translated into a static workflow graph stored in a memory, the static workflow graph comprising a plurality of nodes having input ports and output ports, wherein dependencies between inputs and outputs are specified as edges between input ports and output ports. Information about a first set of nodes is then extracted from the static workflow graph and placed into a dynamic graph. A first actionable job is identified from the dynamic graph and executed.
1. A method of processing a computational workflow, the method comprising:
receiving, by a workflow execution engine executing on a processor, a description of a first computational workflow, the description comprising a plurality of steps, each step having at least one input and at least one output, wherein an input from a second step depends on an output from a first step;
translating the description of the computational workflow into a static workflow graph stored in a memory, the static workflow graph not enabling transformations during runtime and comprising a plurality of nodes having input ports and output ports, wherein dependencies between inputs and outputs are specified as edges between input ports and output ports, the translating further comprising processing the description and creating a node in the static workflow graph for each step in the plurality of steps, wherein steps that specify executable tools are marked as executable nodes and steps that specify containers are marked as container nodes;
extracting, from the static workflow graph, information about a first set of nodes and placing the extracted information into a dynamic workflow graph different from the static graph and enabling transformations during runtime, the extracted information comprising jobs, variables, and links defined by the first set of nodes;
executing a first actionable job from the dynamic workflow graph, the first actionable job comprising a job for which all of its inputs are available;
receiving an indication that the first actionable job has completed, and updating the dynamic workflow graph with new information based on the indication;
determining that additional information is needed from the static workflow graph to continue processing of the computational workflow;
extracting, from the static workflow graph, information about a second set of nodes and placing the extracted information into the dynamic workflow graph;
executing a second actionable job from the dynamic workflow graph, the second actionable job comprising a job for which all of its inputs are available;
selecting a third actionable job from the dynamic workflow graph, the third actionable job comprising at least one of the container nodes;
rolling out the at least one of the container nodes so as to add information therein to the dynamic workflow graph;
executing the third actionable job; and
providing an output to at least one of the nodes downstream of the second set of nodes in the computational workflow without requiring complete execution of the third actionable job.
2. The method of claim 1 , wherein the static workflow graph is fully enumerated from the description of the first computational workflow prior to placing the extracted information into the dynamic workflow graph.
3. The method of claim 1 , wherein the first step references a description of a second computational workflow.
4. The method of claim 3 , wherein the description of the second computational workflow is written in a different format than the description of the first computational workflow.
5. The method of claim 3 , wherein translating the description of the first computational workflow comprises adding the first step as a container node to the static workflow graph, accessing the description of the second computational workflow, and adding any steps described by the second computational workflow into the static workflow graph.
6. The method of claim 1 , wherein translating the description of the first computational workflow further comprises adding every step specified by the description to the static workflow graph.
7. The method of claim 6 , wherein the first set of nodes extracted from the static workflow graph does not include nodes corresponding to steps described by the description of the second computational workflow.
8. The method of claim 1 , wherein the second actionable job comprises an executable.
9. The method of claim 1 , wherein the first actionable job represents an executable described by the first step, and wherein executing the first actionable job comprises submitting the first actionable job to a backend executor.
10. The method of claim 9 , wherein the backend executor is located on a device separate from the workflow execution engine.
11. The method of claim 9 , further comprising:
receiving, from the backend executor, an indication that the first actionable job has completed;
updating the dynamic workflow graph with information related to an output generated by the completed first actionable job;
identifying a second actionable job from the dynamic workflow
graph; and executing the second actionable job.
12. The method of claim 11 , wherein updating the dynamic workflow graph comprises:
updating a value for a variable;
decrementing a port counter for a job related to the updated value; and
determining whether any additional variables may be updated by considering
links
associated with the updated variable.
13. The method of claim 12 , further comprising updating a value for a determined additional variable.
14. The method of claim 1 , wherein the inputs for the first actionable job comprise an array of values, and wherein executing the first actionable job comprises placing a plurality of jobs in the dynamic workflow graph for the first actionable job, wherein each of the placed jobs comprises the executable from the first actionable job and one value of the array of values.
15. The method of claim 14 , wherein the description indicates that the step associated with the first actionable job can be scattered.
16. A system for processing a bioinformatics workflow, the system comprising:
at least one computer hardware processor; and
at least one non-transitory computer-readable storage medium storing processor-executable-instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
receiving, by a workflow execution engine executing on a processor, a description of a first computational workflow, the description comprising a plurality of steps, each step having at least one input and at least one output, wherein an input from a second step depends on an output from a first step;
translating the description of the bioinformatics workflow into a static workflow graph stored in a memory, the static workflow graph not enabling transformations during runtime and comprising a plurality of nodes having input ports and output ports, wherein dependencies between inputs and outputs are specified as edges between input ports and output ports, the translating further comprising processing the description and creating a node in the static workflow graph for each step in the plurality of steps, wherein steps that specify executable tools are marked as executable nodes and steps that specify containers are marked as container nodes;
extracting, from the static workflow graph, information about a first set of nodes and placing the extracted information into a control structure different from the static graph and enabling transformations during runtime, the extracted information comprising variables, jobs, and links defined by the first set of nodes;
executing a first actionable job from the control structure, the first actionable job comprising a job for which all of its inputs are available;
receiving an indication that the first actionable job has completed, and updating variables, jobs, and links in the control structure based on the received indication;
determining that the control structure requires additional information from the static workflow graph to continue processing of the computational workflow;
extracting, from the static workflow graph, information about a second set of nodes and placing the extracted information into the control structure;
executing a second actionable job from the control structure, the second actionable job comprising a job for which all of its inputs are available;
selecting a third actionable job from the dynamic workflow graph, the third actionable job comprising at least one of the container nodes;
rolling out the at least one of the container nodes so as to add information therein to the dynamic workflow graph;
executing the third actionable job; and
providing an output to at least one of the nodes downstream of the second set of nodes in the computational workflow without requiring complete execution of the third actionable job.
17. The system of claim 16 , wherein the first actionable job comprises a container.
18. The system of claim 16 , wherein identifying a first actionable job from the control structure comprises finding jobs having input counters set to zero.
19. The system of claim 16 , wherein the input for the actionable job depends from the output of the first actionable job.
20. The method of claim 1 , wherein determining that the workflow graph requires additional information from the static workflow graph to continue processing of the computational workflow comprises executing an actionable job representing a container node.