Workflow optimization
A computer implemented method, computer program product, and system for managing execution of a workflow comprising a set of subworkflows, comprising optimizing the set of subworkflows using a deep neural network, wherein each subworkflow of the set of subworkflows has a set of tasks, wherein each task of the sets of tasks has a requirement of resources of a set of resources; wherein each task of the sets of tasks is enabled to be dependent on another task of the sets of tasks, training the deep neural network by: executing the set of subworkflows, collecting provenance data from the execution, and collecting monitoring data that represents the state of said set of resources, wherein the training causes the neural network to learn relationships between the states of said set of resources, the said sets of tasks, their parameters and the obtained performance, optimizing an allocation of resources of the set of resources to each task of the sets of tasks to ensure compliance with a user-defined quality metric based on the deep neural network output.
1. A computer implemented method comprising:
executing a workflow using known input data to produce corresponding output data, wherein the workflow comprises a set of subworkflows and each subworkflow of the set of workflows comprises a set of tasks, wherein each task of the set of tasks has a requirement of resources of a set of resources and each task of the set of tasks is enabled to be dependent on one or more other tasks of the sets of tasks;
training a deep neural network based on execution of the set of subworkflows, the training comprising:
collecting provenance data from the execution of the set of subworkflows; and
collecting monitoring data that represents a state of the set of resources;
wherein the training causes the neural network to learn relationships between the states of the set of resources, the sets of tasks, their parameters and an obtained performance, and wherein the relationships between the states of the set of resources, the sets of tasks, their parameters, and the obtained performance are non-linear; and
controlling allocation of resources of the set of resources to each task of the sets of tasks to ensure compliance with a user-defined quality metric based on output of the deep neural network, wherein the optimization further comprises creating at least one non-linear model based on the relationships between the states of the set of resources, the sets of tasks, their parameters, and the obtained performance.
2. The method of claim 1 further comprising:
encoding a set of chromosomes based the output of the deep neural network as part of controlling allocation of the resources of the set of resources to each task of the sets of tasks;
using a Genetic Algorithm, running the set of chromosomes through the deep neural network to determine a new allocation of resources;
determining if the new allocation of resources better complies with a quality of services; and
implementing the new allocation of resources if it does better comply with the quality of services.
3. The method of claim 2 wherein the set of resources includes hardware resources and wherein an amount of hardware resources is enabled to change over execution of the workflows.
4. The method of claim 2 , wherein the workflow is represented by a directed acyclic graph.
5. The method of claim 4 , wherein inputs for the deep neural network are translated into a tensor representation.
6. A computer program product for managing execution of a workflow comprising a set of subworkflows, the computer program product comprising:
a non-transitory computer readable medium encoded with computer executable program code, the code configured to enable the execution of:
executing a workflow using known input data to produce corresponding output data, wherein the workflow comprises a set of subworkflows and each subworkflow of the set of workflows comprises a set of tasks, wherein each task of the set of tasks has a requirement of resources of a set of resources and each task of the set of tasks is enabled to be dependent on one or more other tasks of the sets of tasks;
training a deep neural network based on execution of the set of subworkflows, the training comprising:
collecting provenance data from the execution of the set of subworkflows; and
collecting monitoring data that represents a state of the set of resources;
wherein the training causes the neural network to learn relationships between the states of the set of resources, the sets of tasks, their parameters and an obtained performance, and wherein the relationships between the states of the set of resources, the sets of tasks, their parameters, and the obtained performance are non-linear; and
controlling allocation of resources of the set of resources to each task of the sets of tasks to ensure compliance with a user-defined quality metric based on output of the deep neural network, wherein the optimization further comprises creating at least one non-linear model based on the relationships between the states of the set of resources, the sets of tasks, their parameters, and the obtained performance.
7. The computer program product of claim 6 the code further configured for:
encoding a set of chromosomes based the output of the deep neural network as part of controlling allocation of the resources of the set of resources to each task of the sets of tasks;
using a Genetic Algorithm, running the set of chromosomes through the deep neural network to determine a new allocation of resources;
determining if the new allocation of resources better complies with a quality of services; and
implementing the new allocation of resources if it does better comply with the quality of services.
8. The computer program product of claim 7 , wherein a set of resources includes hardware resources and wherein an amount of hardware resources is enabled to change over execution of the workflows.
9. The computer program product of claim 7 , wherein the workflow is represented by a directed acyclic graph.
10. The computer program product of claim 9 wherein inputs for the deep neural network are translated into a tensor representation.
11. A system for managing execution of a workflow comprising a set of subworkflows, the system comprising:
one or more processors; and
computer executable program code, the code configured to enable the execution across the one or more processors of:
executing a workflow using known input data to produce corresponding output data, wherein the workflow comprises a set of subworkflows and each subworkflow of the set of workflows comprises a set of tasks, wherein each task of the set of tasks has a requirement of resources of a set of resources and each task of the set of tasks is enabled to be dependent on one or more other tasks of the sets of tasks;
training a deep neural network based on execution of the set of subworkflows, the training comprising:
collecting provenance data from the execution of the set of subworkflows; and
collecting monitoring data that represents a state of the set of resources;
wherein the training causes the neural network to learn relationships between the states of the set of resources, the sets of tasks, their parameters and an obtained performance, and wherein the relationships between the states of the set of resources, the sets of tasks, their parameters, and the obtained performance are non-linear; and
controlling allocation of resources of the set of resources to each task of the sets of tasks to ensure compliance with a user-defined quality metric based on output of the deep neural network, wherein the optimization further comprises creating at least one non-linear model based on the relationships between the states of the set of resources, the sets of tasks, their parameters, and the obtained performance.
12. The system of claim 11 the computer executable program code further enabling the execution of:
encoding a set of chromosomes based the output of the deep neural network as part of controlling allocation of the resources of the set of resources to each task of the sets of tasks;
using a Genetic Algorithm, running the set of chromosomes through the deep neural network to determine a new allocation of resources;
determining if the new allocation of resources better complies with the a quality of services; and
implementing the new allocation of resources if it does better comply with the quality of services.
13. The system of claim 12 , wherein the set of resources includes hardware resources and wherein the amount of hardware resources is enabled to change over execution of the workflows.
14. The system of claim 12 , wherein the workflow is represented by a directed acyclic graph.
15. The system of claim 14 , wherein inputs for the deep neural network are translated into a tensor representation.