Snapshots to train prediction models and improve workflow execution

1. A method for managing execution of a given workflow comprising one or more sub-workflows, comprising:

parsing and interpreting a definition of said given workflow and said one or more sub-workflows, wherein said given workflow is one of a plurality of workflows that share at least a portion of resources employed by the given workflow;

flattening said given workflow and said one or more sub-workflows into a directed acyclic graph where nodes represent activities of the given workflow and edges represent partial-order constraints;

collecting provenance data about one or more executions of said given workflow according to said directed acyclic graph, wherein said collected provenance data comprises a resource allocation of said given workflow comprising an assignment of resources to the activities, input parameters of the activities of the given workflow, output parameters of the activities of the given workflow, telemetry metrics indicating the use of one or more computational resources during the execution of one or more of said activities of the activities of the given workflow and data regarding an execution of additional workflows in the plurality of workflows, wherein the data for the additional workflows comprises one or more features of the additional workflows and an indication of one or more activities of the additional workflows;

taking one or more snapshots of the collected provenance data as a representation of a state of one or more prior executions of the given workflow, wherein the one or more snapshots comprise a selection of features from said input and output parameters, and said telemetry metrics indicating the use of the one or more computational resources at the time of the snapshot;

calculating, using at least a portion of the collected provenance data, one or more quality metrics regarding one or more prior executions of the plurality of workflows;

associating said one or more snapshots to said one or more quality metrics;

training one or more prediction models of said quality metrics using machine learning techniques based on said one or more snapshots of a plurality of said prior executions of the plurality of workflows; and

processing said one or more prediction models during a real-time execution of said given workflow to adjust the execution of said given workflow of the plurality of workflows, by updating the input parameters of said activities of the given workflow and the resource allocation of said given workflow, taking into account one or more of said quality metrics for the plurality of workflows predicted by said one or more prediction models,

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 , wherein said step of taking one or more snapshots is triggered when a given execution crosses one or more elements of said directed acyclic graph that are marked with a predefined flag.

3. The method of claim 1 , wherein said step of taking one or more snapshots is triggered at a specified time interval during a given execution.

4. The method of claim 1 , wherein the one or more prediction models are updated substantially continuously as more provenance data about new executions are collected.

5. The method of claim 1 , wherein the one or more snapshots included in the step of training one or more prediction models comprise one or more snapshots taken during the one or more executions of said given workflow and one or more additional snapshots generated after a completion of the workflow executions.

6. The method of claim 1 , wherein said collected provenance data comprises one or more provenance data resulting from historical executions of the workflow.

7. A computer program product for managing execution of a given workflow comprising one or more sub-workflows, comprising a non-transitory machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps:

parsing and interpreting a definition of said given workflow and said one or more sub-workflows, wherein said given workflow is one of a plurality of workflows that share at least a portion of resources employed by the given workflow;

flattening said given workflow and said one or more sub-workflows into a directed acyclic graph where nodes represent activities of the given workflow and edges represent partial-order constraints;

collecting provenance data about one or more executions of said given workflow according to said directed acyclic graph, wherein said collected provenance data comprises a resource allocation of said given workflow comprising an assignment of resources to the activities, input parameters of the activities of the given workflow, output parameters of the activities of the given workflow, telemetry metrics indicating the use of one or more computational resources during the execution of one or more of said activities of the activities of the given workflow and data regarding an execution of additional workflows in the plurality of workflows, wherein the data for the additional workflows comprises one or more features of the additional workflows and an indication of one or more activities of the additional workflows;

taking one or more snapshots of the collected provenance data as a representation of a state of one or more prior executions of the given workflow, wherein the one or more snapshots comprise a selection of features from said input and output parameters, and said telemetry metrics indicating the use of the one or more computational resources at the time of the snapshot;

calculating, using at least a portion of the collected provenance data, one or more quality metrics regarding one or more prior executions of the plurality of workflows;

associating said one or more snapshots to said one or more quality metrics;

training one or more prediction models of said quality metrics using machine learning techniques based on said one or more snapshots of a plurality of said prior executions of the plurality of workflows; and

processing said one or more prediction models during a real-time execution of said given workflow to adjust the execution of said given workflow of the plurality of workflows, by updating the input parameters of said activities of the given workflow and the resource allocation of said given workflow, taking into account one or more of said quality metrics for the plurality of workflows predicted by said one or more prediction models.

8. The computer program product of claim 7 , wherein said step of taking one or more snapshots is triggered by one or more of when a given execution crosses one or more elements of said directed acyclic graph that are marked with a predefined flag and at a specified time interval during a given execution.

9. The computer program product of claim 7 , wherein the one or more prediction models are updated substantially continuously as more provenance data about new executions are collected.

10. The computer program product of claim 7 , wherein the one or more snapshots included in the step of training one or more prediction models comprise one or more snapshots taken during the one or more executions of said given workflow and one or more additional snapshots generated after a completion of the workflow executions.

11. The computer program product of claim 7 , wherein said collected provenance data comprises one or more provenance data resulting from historical executions of the workflow.

12. A system for managing execution of a given workflow comprising one or more sub-workflows, comprising:

a memory; and

at least one processing device, coupled to the memory, operative to implement the following steps:

parsing and interpreting a definition of said given workflow and said one or more sub-workflows, wherein said given workflow is one of a plurality of workflows that share at least a portion of resources employed by the given workflow;

flattening said given workflow and said one or more sub-workflows into a directed acyclic graph where nodes represent activities of the given workflow and edges represent partial-order constraints;

collecting provenance data about one or more executions of said given workflow according to said directed acyclic graph, wherein said collected provenance data comprises a resource allocation of said given workflow comprising an assignment of resources to the activities, input parameters of the activities of the given workflow, output parameters of the activities of the given workflow, telemetry metrics indicating the use of one or more computational resources during the execution of one or more of said activities of the activities of the given workflow and data regarding an execution of additional workflows in the plurality of workflows, wherein the data for the additional workflows comprises one or more features of the additional workflows and an indication of one or more activities of the additional workflows;

taking one or more snapshots of the collected provenance data as a representation of a state of one or more prior executions of the given workflow, wherein the one or more snapshots comprise a selection of features from said input and output parameters, and said telemetry metrics indicating the use of the one or more computational resources at the time of the snapshot;

calculating, using at least a portion of the collected provenance data, one or more quality metrics regarding one or more prior executions of the plurality of workflows;

associating said one or more snapshots to said one or more quality metrics;

training one or more prediction models of said quality metrics using machine learning techniques based on said one or more snapshots of a plurality of said prior executions of the plurality of workflows; and

processing said one or more prediction models during a real-time execution of said given workflow to adjust the execution of said given workflow of the plurality of workflows, by updating the input parameters of said activities of the given workflow and the resource allocation of said given workflow, taking into account one or more of said quality metrics for the plurality of workflows predicted by said one or more prediction models.

13. The system of claim 12 , wherein said step of taking one or more snapshots is triggered when a given execution crosses one or more elements of said directed acyclic graph that are marked with a predefined flag.

14. The system of claim 12 , wherein said step of taking one or more snapshots is triggered at a specified time interval during a given execution.

15. The system of claim 12 , wherein the one or more prediction models are updated substantially continuously as more provenance data about new executions are collected.

16. The system of claim 12 , wherein the one or more snapshots included in the step of training one or more prediction models comprise one or more snapshots taken during the one or more executions of said given workflow and one or more additional snapshots generated after a completion of the workflow executions.

17. The system of claim 12 , wherein said collected provenance data comprises one or more provenance data resulting from historical executions of the workflow.

18. The method of claim 1 , wherein each of the plurality of workflows are executed at a same time.

19. The computer program product of claim 7 , wherein each of the plurality of workflows are executed at a same time.

20. The system of claim 12 , wherein each of the plurality of workflows are executed at a same time.