Automatic creation and execution of a test harness for workflows
View Patent ↗A computer implemented method for automatically creating and executing a test harness for a workflow includes creating a workflow package including a version identification and a workflow graph definition and creating a workflow instance as a copy of the workflow package. The workflow instance is executed to generate output data and the output data is saved into the workflow instance. The workflow instance is saved, with the output data, as a test archive. One or more components of an updated workflow package are tested with the test archive; and an indication of pass or fail of the testing of the updated workflow package is outputted.
1. A computer implemented method, comprising:
automatically creating and executing a test harness for an existing software program workflow, the creating and executing comprising:
receiving the exiting software program workflow, wherein the existing software program workflow comprises software components;
automatically creating, by a computing device, a workflow package of a direct acyclic graph (DAG), including a version identification and a workflow graph definition based on the existing workflow;
creating a workflow instance based on the workflow package;
executing the workflow instance on the computing device to generate output data;
saving the workflow instance with the generated output data as a test archive, wherien the generated output data is used as a baseline for an output comparison test;
augmenting a file structure of the workflow instance, by the test archive, by inserting an input data and the generated output data used as a baseline for the output comparison test, and at least one rule, on one or more tests to be executed on one or more software components in an updated workflow package; and
testing the one or more softwaqre components of the updated workflow package with the test archive by performing the output comparison test; and
output an indication of pass or fail of the testing of the updated workflow package based on the testing.
2. The computer implemented method of claim 1 , further comprising creating the workflow package including information on a transformation logic for each node.
3. The computer implemented method of claim 1 , further comprising generating the output data of the workflow instance via at least one of (a) executing one or more selected subgraphs from a plurality of subgraphs of the workflow to generate output data, and (b) populating other ones of the plurality of subgraphs with subgraph output data from a previously created workflow instance.
4. The computer implemented method of claim 1 , wherein the indication includes results from each test performed on each of the one or more components.
5. The computer implemented method of claim 1 , wherein the testing includes a plurality of tests comprising an input data test, a data staging test, an execution test, and an output comparison test.
6. The computer implemented method of claim 5 , wherein each of the plurality of tests are executed in order, where failure of one of the plurality of tests ends the testing and outputs a failure indication.
7. The computer implemented method of claim 1 , further comprising associating a test specification with the test archive, wherein the test specification defines which output data from the one or more components should be added to the test archive.
8. The computer implemented method of claim 1 , further comprising associating a test specification with the test archive, wherein the test specification permits customization of the testing and provides definitions of pass and fail.
9. The computer implemented method of claim 1 , further comprising permitting additional programs to be provided in a test specification for a selected testing of the one or more components.
10. The computer implemented method of claim 1 , further comprising regenerating the test archive based on an instance of the updated workflow package.
11. A computer implemented method of claim 1 , wherein:
creating a workflow package of the direct acyclic graph (DAG) includes a transformation logic of each component of one or more components of the workflow;
the testing includes a plurality of tests comprising an input data test, a data staging test, an execution test, and an output comparison test; and
each of the plurality of tests are executed in order, where failure of one of the plurality of tests ends the testing.
12. The computer implemented method of claim 11 , wherein the indication includes results from each test performed on each of the one or more components.
13. The computer implemented method of claim 11 , further comprising associating a test specification with the test archive, wherein the test specification defines which output data from the one or more components should be added to the test archive.
14. The computer implemented method of claim 11 , further comprising associating a test specification with the test archive, wherein the test specification permits customization of the testing and provides definitions of pass and fail.
15. The computer implemented method of claim 11 , further comprising regenerating the test archive based on an instance of the updated workflow package.
16. The computer implemented method of claim 1 , further comprising creating a separate workflow instance for each software component.
17. A non-transitory computer readable storage medium tangibly embodying a computer readable program code having computer readable instructions that, when executed, causes a computer device to carry out a method:
automatically creating and executing a test harness for an existing software program workflow, the creating and executing comprising:
receiving the existing software program workflow, wherein the existing software program workflow comprises software components;
automatically creating, by a computing device, a workflow package of a direct acyclic graph (DAG); including a version identificaiton and a workflow graph definition based on the existing workflow;
creating a workflow instance based on the workflow package;
executing the workflow instance on the vcomputing device to generate output data;
saving the workflow instance with the generated output data as a test archive, wherein the generated output data is used as a baseline for an output comparison test;
augmenting a file structure of the workflow instance, by the test archive, by inserting an input data and the generated output data used as a baseline for the output cmparison test, and at least one rule, on one or more tests to be executed on one or more software components in an updated workflow package; and
testing the one or more software components of the updated workflow package with the test archive by performing the output comparison test; and
outputting an indication of pass of fail of the tsting of the updated workflow package based on the testing.
18. The non-transitory computer readable storage medium of claim 17 , wherein the testing includes a plurality of tests comprising an input data test, a data staging test, an execution test, and an output comparison test.
19. The non-transitory computer readable storage medium of claim 18 , wherein if one of the plurality of tests fail for a given component, the indication provided is a failure indication for both the given component and the workflow.
20. The non-transitory computer readable storage medium of claim 18 , wherein each of the plurality of tests are executed in order, where failure of one of the plurality of tests ends the testing and outputs a failure indication.
21. The non-transitory computer readable storage medium of claim 17 , wherein the execution of the code by the processor further configures the computing device to perform an act comprising associating a test specification with the test archive, wherein the test specification defines which of output data from the one or more components should be added to the test archive.
22. The non-transitory computer readable storage medium of claim 17 , wherein the execution of the code by the processor further configures the computing device to perform an act comprising associating a test specification with the test archive, wherein the test specification permits customization of the testing and provides definitions of pass and fail.
23. The non-transitory computer readable storage medium of claim 17 , wherein:
creating a workflow package of the direct acyclic graph (DAG) includes a transformation logic of each component of one or more components of the workflow;
the testing includes a plurality of tests comprising an input data test, a data staging test, an execution test, and an output comparison test; and
each of the plurality of tests are executed in order, where failure of one of the plurality of tests ends the testing.
24. The non-transitory computer readable storage medium of claim 23 , wherein the execution of the code by the processor further configures the computing device to perform an act comprising associating a test specification with the test archive, wherein the test specification defines which of output data from the one or more components should be added to the test archive.
25. The non-transitory computer readable storage medium of claim 23 , wherein the execution of the code by the processor further configures the computing device to perform an act associating a test specification with the test archive, wherein the test specification permits customization of the testing and provides definitions of pass and fail.
26. The non-transitory computer readable storage medium of claim 23 , wherein the execution of the code by the processor further configures the computing device to perform an act comprising regenerating the test archive based on an instance of the updated workflow package.