Systems and methods for processing computational workflows
In one embodiment, a method of processing a computational workflow comprises receiving a description of a computational workflow. The description comprises a plurality of steps, in which each step has at least one input and at least one output, and further wherein an input from a second step depends on an output from a first step. The description is translated into a static workflow graph stored in a memory, the static workflow graph comprising a plurality of nodes having input ports and output ports, wherein dependencies between inputs and outputs are specified as edges between input ports and output ports. Information about a first set of nodes is then extracted from the static workflow graph and placed into a dynamic graph. A first actionable job is identified from the dynamic graph and executed.
1. A method of processing a computational workflow, the method comprising:
receiving, by a workflow execution engine executing on a processor, a description of a first computational workflow, the description comprising a plurality of steps;
translating the description of the computational workflow into a static workflow graph stored in a memory, the static workflow graph not enabling transformations during runtime and comprising a plurality of nodes, the translating further comprising processing the description and creating a node in the static workflow graph for each step in the plurality of steps, wherein steps that specify executable tools are marked as executable nodes and steps that specify containers are marked as container nodes;
extracting, from the static workflow graph, information about a first set of nodes and placing the extracted information into a dynamic workflow graph different from the static graph and enabling transformations during runtime;
selecting a first actionable job from the dynamic workflow graph;
upon determining that the selected first actionable job represents one of the executable nodes, (i) executing the selected first actionable job, (ii) receiving an indication that the selected first actionable job has completed, and (iii) updating the dynamic workflow graph with new information based on the indication; and
upon determining that the selected first actionable job represents one of the container nodes, (i) rolling out all first-level nodes within the one of the container nodes, (ii) identifying one or more new actionable jobs as a result of the rollout, and (iii) updating the dynamic workflow graph based on the identified one or more new actionable jobs,
wherein the first-level nodes comprise at least one of the executable nodes or the container nodes but not contents of the container nodes.
2. The method of claim 1 , wherein the static workflow graph is fully enumerated from the description of the first computational workflow prior to placing the extracted information into the dynamic workflow graph.
3. The method of claim 1 , wherein at least one of the steps references a description of a second computational workflow.
4. The method of claim 3 , wherein the description of the second computational workflow is written in a different format than the description of the first computational workflow.
5. The method of claim 3 , wherein translating the description of the first computational workflow comprises adding the at least one of the steps as a container node to the static workflow graph, accessing the description of the second computational workflow, and adding any steps described by the second computational workflow into the static workflow graph.
6. The method of claim 1 , wherein translating the description of the first computational workflow further comprises adding every step specified by the description to the static workflow graph.
7. The method of claim 6 , wherein the first set of nodes extracted from the static workflow graph does not include nodes corresponding to steps described by the description of the second computational workflow.
8. The method of claim 1 , wherein the selected first actionable job represents an executable described by the at least one of the steps, and wherein executing the selected first actionable job comprises submitting the first actionable job to a backend executor.
9. The method of claim 8 , wherein the backend executor is located on a device separate from the workflow execution engine.
10. The method of claim 8 , further comprising:
identifying a second actionable job from the dynamic workflow graph; and
executing the second actionable job.
11. The method of claim 1 , wherein updating the dynamic workflow graph comprises:
updating a value for a variable;
decrementing a port counter for a job related to the updated value; and
determining whether any additional variables may be updated by considering links associated with the updated variable.
12. The method of claim 11 , further comprising updating a value for a determined additional variable.
13. The method of claim 1 , wherein inputs for the selected first actionable job comprise an array of values, and wherein executing the selected first actionable job comprises placing a plurality of jobs in the dynamic workflow graph for the selected first actionable job, wherein each of the placed jobs comprises the executable from the selected first actionable job and one value of the array of values.
14. The method of claim 13 , wherein the description indicates that one of the plurality steps associated with the selected first actionable job can be scattered.
15. A system for processing a bioinformatics workflow, the system comprising:
at least one computer hardware processor; and
at least one non-transitory computer-readable storage medium storing processor-executable-instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:
receiving, by a workflow execution engine executing on a processor, a description of a first computational workflow, the description comprising a plurality of steps;
translating the description of the bioinformatics workflow into a static workflow graph stored in a memory, the static workflow graph not enabling transformations during runtime and comprising a plurality of nodes, the translating further comprising processing the description and creating a node in the static workflow graph for each step in the plurality of steps, wherein steps that specify executable tools are marked as executable nodes and steps that specify containers are marked as container nodes;
extracting, from the static workflow graph, information about a first set of nodes and placing the extracted information into a control structure different from the static graph and enabling transformations during runtime;
selecting a first actionable job from the control structure;
upon determining that the selected first actionable job represents one of the executable nodes, (i) executing the selected first actionable job, (ii) receiving an indication that the selected first actionable job has completed, and (iii) updating the control structure with new information based on the received indication;
upon determining that the selected first actionable job represents one of the container nodes, (i) rolling out all first-level nodes within the one of the container nodes, (ii) identifying one or more new actionable jobs as a result of the roll out, and (iii) updating the dynamic workflow graph based on the identified one or more new actionable jobs,
wherein the first-level nodes comprise at least one of the executable nodes or the container nodes but not contents of the container nodes.
16. The system of claim 15 , wherein the selected first actionable job has input counters set to zero.
17. The system of claim 15 , wherein inputs for the selected first actionable job comprise an array of values and executing the selected first actionable job comprises placing a plurality of jobs in the dynamic workflow graph for the selected first actionable job, each of the placed jobs comprising the executable from the selected first actionable job and one value of the array of values.
18. The system of claim 15 , wherein the static workflow graph is fully enumerated from the description of the first computational workflow prior to placing the extracted information into the dynamic workflow graph.
19. The system of claim 15 , wherein at least one of the steps references a description of a second computational workflow.
20. The system of claim 19 , wherein the description of the second computational workflow is written in a different format than the description of the first computational workflow.