Automated runtime configuration for dataflows
Methods, systems and computer program products are provided for automated runtime configuration for dataflows to automatically select or adapt a runtime environment or resources to a dataflow plan prior to execution. Metadata generated for dataflows indicates dataflow information, such as numbers and types of sources, sinks and operations, and the amount of data being consumed, processed and written. Weighted dataflow plans are created from unweighted dataflow plans based on metadata. Weights that indicate operation complexity or resource consumption are generated for data operations. A runtime environment or resources to execute a dataflow plan is/are selected based on the weighted dataflow and/or a maximum flow. Preferences may be provided to influence weighting and runtime selections.
1. A method in a server, comprising:
receiving a dataflow plan comprising representations of data operations in a dataflow pipeline;
determining, based on the received dataflow plan, metadata of the received dataflow plan;
determining, based on the received dataflow plan, a feature set comprising a plurality of features;
selecting a machine learning (ML) model from a plurality of ML models based on the metadata of the received dataflow plan and metadata of a training set the selected ML model was trained on;
providing the feature set to the selected ML model;
receiving, from the selected ML model, the weighted dataflow plan associating a weight associated with a data operation of the data operations; and
causing execution of the received dataflow plan by resources allocated based on the weight.
2. The method of claim 1 , wherein the weight is selected from a weight range corresponding to an execution complexity range for the data operation in terms of consumption of at least one of computing, memory, storage or network resources.
3. The method of claim 1 , wherein the weight comprises a combined weight or a weight vector indicating at least two of computing, memory, storage, and network resources.
4. The method of claim 1 , wherein the ML model determines the weight based at least on a portion of the metadata of the received dataflow plan corresponding to the data operation.
5. The method of claim 1 , further comprising:
determining a maximum flow for the dataflow plan having the weight applied thereto,
wherein said causing execution of the received dataflow plan comprises:
causing allocation of the resources based on the maximum flow.
6. The method of claim 5 , wherein said determining the maximum dataflow comprises:
applying a max-flow min-cut theorem to the dataflow plan having the weight applied thereto.
7. The method of claim 1 , wherein said causing execution of the received dataflow plan comprises:
causing allocation of the resources based on an indication of a preference related to at least one of execution time to execute the received dataflow plan or execution cost to execute the received dataflow plan.
8. The method of claim 1 , wherein said causing execution of the received dataflow plan comprises:
providing feedback to a user interface indicating a plurality of execution environments and associated costs; and
prompting user input to select an execution environment from among the plurality of execution environments to execute the received dataflow plan.
9. The method of claim 1 , wherein said causing execution of the received dataflow plan comprises:
providing the weight to a second ML model; and
causing allocation of the resources based at least on an output generated by the second ML model based at least on the weight.
10. A system, comprising:
a processor; and
memory that stores program code executable by the processor to perform a method, the method comprising:
receiving a dataflow plan comprising representations of data operations in a dataflow pipeline;
determining, based on the received dataflow plan, metadata of the received dataflow plan;
determining, based on the received dataflow plan, a feature set comprising a plurality of features;
selecting a machine learning (ML) model from a plurality of ML models based on metadata of the received dataflow plan and metadata of a training set the selected ML model was trained on;
providing the feature set to the selected ML model;
receiving, from the selected ML model, a weight associated with a data operation of the data operations; and
causing execution of the received dataflow plan by resources allocated based on the weight.
11. The system of claim 10 , wherein the weight is selected from a weight range corresponding to an execution complexity range for the data operation in terms of consumption of at least one of computing, memory, storage or network resources.
12. The system of claim 10 , wherein the weight comprises a combined weight or a weight vector indicating at least two of computing, memory, storage, and network resources.
13. The system of claim 10 , wherein the ML model determines the weight based at least on the corresponding metadata.
14. The system of claim 10 , wherein the method further comprises:
determining a maximum flow for the dataflow plan having the weight applied thereto,
wherein said causing execution of the received dataflow plan comprises:
causing allocation of the resources based on the maximum flow.
15. The system of claim 14 , wherein said determining the maximum dataflow comprises:
applying a max-flow min-cut theorem to the dataflow plan having the weight applied thereto.
16. The system of claim 10 , wherein said causing execution of the received dataflow plan comprises:
causing allocation of the resources based on an indication of a preference related to at least one of execution time to execute the received dataflow plan or execution cost to execute the received dataflow plan.
17. The system of claim 10 , wherein said causing execution of the received dataflow plan comprises:
providing feedback to a user interface indicating a plurality of execution environments and associated costs; and
prompting user input to select an execution environment from among the plurality of execution environments to execute the received dataflow plan.
18. The system of claim 10 , wherein said causing execution of the received dataflow plan comprises:
providing the weight to a second ML model; and
causing allocation of the resources based at least on an output generated by the second ML model based at least on the weight.
19. A method in a server, comprising:
receiving a dataflow plan comprising representations of data operations in a dataflow pipeline;
determining, based on the received dataflow plan, metadata of the received dataflow plan that comprises information regarding the data operations;
selecting a machine learning (ML) model from a plurality of ML models based on the metadata of the dataflow plan and metadata of a training set the selected ML model was trained on;
providing the generated metadata to the selected ML model;
receiving, from the selected ML model, a weight associated with a data operation of the data operations; and
causing execution of the received dataflow plan by resources allocated based on the weight.
20. The method of claim 19 , wherein said causing execution of the received dataflow plan comprises:
determining a maximum flow for the dataflow plan having the weight applied thereto; and
causing allocation of the resources based on the maximum flow.