Workflow optimization
A computer implemented method, computer program product, and system for managing execution of a workflow comprising a set of subworkflows, comprising optimizing the set of subworkflows using a deep neural network, wherein each subworkflow of the set of subworkflows has a set of tasks, wherein each task of the sets of tasks has a requirement of resources of a set of resources; wherein each task of the sets of tasks is enabled to be dependent on another task of the sets of tasks, training the deep neural network by: executing the set of subworkflows, collecting provenance data from the execution, and collecting monitoring data that represents the state of said set of resources, wherein the training causes the neural network to learn relationships between the states of said set of resources, the said sets of tasks, their parameters and the obtained performance, optimizing an allocation of resources of the set of resources to each task of the sets of tasks to ensure compliance with a user-defined quality metric based on the deep neural network output.
1. A computer implemented method for managing execution of a workflow comprising a set of subworkflows, the method comprising:
optimizing the set of subworkflows using a deep neural network; wherein each subworkflow of the set of subworkflows has a set of tasks; wherein each task of the sets of tasks has a requirement of resources of a set of resources; wherein each task of the sets of tasks is enabled to be dependent on another task of the sets of tasks;
training the deep neural network by: executing the set of subworkflows, collecting provenance data from the execution, and collecting monitoring data that represents a state of the set of resources; wherein the training causes the neural network to learn relationships between the states of the set of resources, the sets of tasks, their parameters and an obtained performance, and wherein the relationships between the states of the set of resources, the sets of tasks, their parameters, and the obtained performance are non-linear; and
optimizing an allocation of resources of the set of resources to each task of the sets of tasks to ensure compliance with a user-defined quality metric based on output of the deep neural network, wherein the optimization further comprises creating at least one non-linear model based on the relationships between the states of the set of resources, the sets of tasks, their parameters, and the obtained performance.
2. The method of claim 1 further comprising:
using the optimized allocation of resources to encode a set of chromosomes;
using a Genetic Algorithm, running the set of chromosomes through the deep neural network to determine a new allocation of resources;
determining if the new allocation of resources better complies with a quality of services; and
implementing the new allocation of resources if it does better comply with the quality of services.
3. The method of claim 2 wherein the set of resources includes hardware resources and wherein an amount of hardware resources is enabled to change over execution of the workflows.
4. The method of claim 2 wherein the workflow is represented by a directed acyclic graph.
5. The method of claim 4 wherein inputs for the deep neural network are translated into a tensor representation.
6. A computer program product for managing execution of a workflow comprising a set of subworkflows, the computer program product comprising:
a non-transitory computer readable medium encoded with computer executable program code, the code configured to enable the execution of:
optimizing the set of subworkflows using a deep neural network; wherein each subworkflow of the set of subworkflows has a set of tasks; wherein each task of the sets of tasks has a requirement of resources of a set of resources; wherein each task of the sets of tasks is enabled to be dependent on another task of the sets of tasks;
training the deep neural network by: executing the set of subworkflows, collecting provenance data from the execution, and collecting monitoring data that represents a state of the set of resources; wherein the training causes the neural network to learn relationships between the states of the set of resources, the sets of tasks, their parameters and an obtained performance, and wherein the relationships between the states of the set of resources, the sets of tasks, their parameters, and the obtained performance are non-linear; and
optimizing an allocation of resources of the set of resources to each task of the sets of tasks to ensure compliance with a user-defined quality metric based on output of the deep neural network, wherein the optimization further comprises creating at least one non-linear model based on the relationships between the states of the set of resources, the sets of tasks, their parameters, and the obtained performance.
7. The computer program product of claim 6 the code further configured for:
using the optimized allocation of resources to encode a set of chromosomes;
using a Genetic Algorithm, running the set of chromosomes through the deep neural network to determine a new allocation of resources;
determining if the new allocation of resources better complies with a quality of services; and
implementing the new allocation of resources if it does better comply with the quality of services.
8. The computer program product of claim 7 wherein a set of resources includes hardware resources and wherein an amount of hardware resources is enabled to change over execution of the workflows.
9. The computer program product of claim 7 wherein the workflow is represented by a directed acyclic graph.
10. The computer program product of claim 9 wherein inputs for the deep neural network are translated into a tensor representation.
11. A system for managing execution of a workflow comprising a set of subworkflows, the system comprising:
one or more processors; and
computer executable program code, the code configured to enable the execution across the one or more processors of:
optimizing the set of subworkflows using a deep neural network; wherein each subworkflow of the set of subworkflows has a set of tasks; wherein each task of the sets of tasks has a requirement of resources of a set of resources; wherein each task of the sets of tasks is enabled to be dependent on another task of the sets of tasks;
training the deep neural network by: executing the set of subworkflows, collecting provenance data from the execution, and collecting monitoring data that represents a state of the set of resources; wherein the training causes the neural network to learn relationships between the states of the set of resources, the sets of tasks, their parameters and an obtained performance, and wherein the relationships between the states of the set of resources, the sets of tasks, their parameters, and the obtained performance are non-linear; and
optimizing an allocation of resources of the set of resources to each task of the sets of tasks to ensure compliance with a user-defined quality metric based on output of the deep neural network, wherein the optimization further comprises creating at least one non-linear model based on the relationships between the states of the set of resources, the sets of tasks, their parameters, and the obtained performance.
12. The system of claim 11 the computer executable program code further enabling the execution of:
using the optimized allocation of resources to encode a set of chromosomes;
using a Genetic Algorithm, running the set of chromosomes through the deep neural network to determine a new allocation of resources;
determining if the new allocation of resources better complies with the a quality of services; and
implementing the new allocation of resources if it does better comply with the quality of services.
13. The system of claim 12 wherein the set of resources includes hardware resources and wherein the amount of hardware resources is enabled to change over execution of the workflows.
14. The system of claim 12 wherein the workflow is represented by a directed acyclic graph.
15. The system of claim 14 wherein inputs for the deep neural network are translated into a tensor representation.
16. A computer implemented method, comprising:
optimizing a workflow hierarchy using a deep neural network, wherein the workflow hierarchy comprises at least one workflow having a corresponding set of subworkflows; wherein each subworkflow of the set of subworkflows has a set of tasks; wherein each task of the sets of tasks has a requirement of resources of a set of resources; wherein each task of the sets of tasks is enabled to be dependent on another task of the sets of tasks;
training the deep neural network by: executing the set of subworkflows, collecting provenance data from the execution, and collecting monitoring data that represents a state of the set of resources; wherein the training causes the neural network to learn relationships between the states of the set of resources, the sets of tasks, their parameters and an obtained performance, and wherein the relationships between the states of the set of resources, the sets of tasks, their parameters, and the obtained performance are non-linear; and
optimizing an allocation of resources of the set of resources to each task of the sets of tasks to ensure compliance with a user-defined quality metric based on output of the deep neural network, wherein the optimization further comprises creating at least one non-linear model based on the relationships between the states of the set of resources, the sets of tasks, their parameters, and the obtained performance.
17. The method of claim 16 further comprising:
using the optimized allocation of resources to encode a set of chromosomes;
using a Genetic Algorithm, running the set of chromosomes through the deep neural network to determine a new allocation of resources;
determining if the new allocation of resources better complies with a quality of services; and
implementing the new allocation of resources if it does better comply with the quality of services.
18. The method of claim 17 wherein the set of resources includes hardware resources and wherein an amount of hardware resources is enabled to change over execution of the workflows.
19. The method of claim 17 wherein the workflow is represented by a directed acyclic graph.
20. The method of claim 19 wherein inputs for the deep neural network are translated into a tensor representation.