Dynamically adapting task execution parallelism of distributed applications

1 . A computer-implemented method for scheduling tasks for concurrent execution in parallel, the computer-implemented method comprising:

receiving, by a computer, from a set of worker nodes via a network, metrics corresponding to execution of tasks in a plurality of stages of a distributed application that are running on the set of worker nodes;

performing, by the computer, using a trained prediction model, an analysis of the metrics corresponding to the execution of the tasks in the plurality of stages of the distributed application received from the set of worker nodes;

determining, by the computer, using the trained prediction model, an optimal per-stage task execution parallelism level for each respective stage of the plurality of stages based on the analysis of the metrics corresponding to the execution of the tasks in the plurality of stages of the distributed application;

scheduling, by the computer, the tasks to execute on the set of worker nodes based on the optimal per-stage task execution parallelism level determined for each respective stage of the plurality of stages to improve at least one of runtime of the distributed application and resource utilization by the distributed application;

updating, by the computer, using the trained prediction model, the optimal per-stage task execution parallelism level of each respective stage of the plurality of stages of the distributed application after a first run of the distributed application to form an updated optimal per-stage task execution parallelism level based on the metrics corresponding to execution of a set of tasks in each respective stage of the plurality of stages of the distributed application collected from the set of worker nodes during the first run of the distributed application;

utilizing, by the computer, the updated optimal per-stage task execution parallelism level to schedule the set of tasks in each respective stage of the plurality of stages of the distributed application on the set of worker nodes during subsequent runs of the distributed application;

collecting, by the computer, historic metrics during at least one of the subsequent runs of the distributed application and runs of a set of similar distributed applications; and

performing, by the computer, using the trained prediction model, a subsequent update of the updated optimal per-stage task execution parallelism level of each respective stage of the plurality of stages of the distributed application based on the historic metrics collected during the at least one of the subsequent runs of the distributed application and the runs of the set of similar distributed applications, wherein the computer determines similarity between the distributed application and the set of similar distributed applications by comparing one or more of anonymized execution graphs, data dependencies, and binaries corresponding to the distributed application and each of the set of similar distributed applications.

2 . The computer-implemented method of claim 1 , further comprising:

running, by the computer, a set of artificial workloads on the distributed application;

generating, by the computer, generic metrics corresponding to the execution of the set of tasks in each respective stage of the plurality of stages of the distributed application based on running the artificial workloads on the distributed application; and

generating, by the computer, a fingerprint for each respective stage of the plurality of stages of the distributed application based on the generic metrics corresponding to the execution of the set of tasks in each respective stage of the plurality of stages of the distributed application.

3 . The computer-implemented method of claim 2 , further comprising:

training, by the computer, a prediction model initially offline to form the trained prediction model using the fingerprint of each respective stage of the plurality of stages of the distributed application that was based on the generic metrics corresponding to the execution of the set of tasks in each respective stage of the plurality of stages; and

determining, by the computer, using the trained prediction model, the optimal per-stage task execution parallelism level of each respective stage of the plurality of stages of the distributed application for the first run of the distributed application.

4 . The computer-implemented method of claim 3 , further comprising:

scheduling, by the computer, the set of tasks of each respective stage of the plurality of stages of the distributed application to execute on the set of worker nodes during the first run of the distributed application based on the optimal per-stage task execution parallelism level determined for each respective stage of the plurality of stages; and

collecting, by the computer, the metrics corresponding to the execution of the set of tasks in each respective stage of the plurality of stages of the distributed application from the set of worker nodes during the first run of the distributed application.

5 . The computer-implemented method of claim 1 , wherein the computer schedules tasks of a same stage on a same worker node to reduce cold starts, and wherein the computer schedules tasks of two or more concurrent stages in parallel such that each of the two or more concurrent stages are estimated to complete at a same time avoiding straggler stage delays.

6 . The computer-implemented method of claim 1 , wherein the metrics include at least one of a number of tasks in a stage, average runtime of each task in the stage, runtime variance of each task in the stage based on level of correlation to input data size, task runtime on warm executors versus task runtime on cold executors, amount of input and output data read or written by each task in the stage, amount of intermediary data read or written by each task in the stage, amount of memory consumed, amount of data spilled to secondary storage, amount of input and output variance by each task in the stage, and time needed to load shared data from a shared data cache.

7 . A computer system for scheduling tasks for concurrent execution in parallel, the computer system comprising:

a communication fabric;

a storage device connected to the communication fabric, wherein the storage device stores program instructions; and

a processor connected to the communication fabric, wherein the processor executes the program instructions to:

receive, from a set of worker nodes via a network, metrics corresponding to execution of tasks in a plurality of stages of a distributed application that are running on the set of worker nodes;

perform, using a trained prediction model, an analysis of the metrics corresponding to the execution of the tasks in the plurality of stages of the distributed application received from the set of worker nodes;

determine, using the trained prediction model, an optimal per-stage task execution parallelism level for each respective stage of the plurality of stages based on the analysis of the metrics corresponding to the execution of the tasks in the plurality of stages of the distributed application;

schedule the tasks to execute on the set of worker nodes based on the optimal per-stage task execution parallelism level determined for each respective stage of the plurality of stages to improve at least one of runtime of the distributed application and resource utilization by the distributed application;

update, using the trained prediction model, the optimal per-stage task execution parallelism level of each respective stage of the plurality of stages of the distributed application after a first run of the distributed application to form an updated optimal per-stage task execution parallelism level based on the metrics corresponding to execution of a set of tasks in each respective stage of the plurality of stages of the distributed application collected from the set of worker nodes during the first run of the distributed application;

utilize the updated optimal per-stage task execution parallelism level to schedule the set of tasks in each respective stage of the plurality of stages of the distributed application on the set of worker nodes during subsequent runs of the distributed application;

collect historic metrics during at least one of the subsequent runs of the distributed application and runs of a set of similar distributed applications; and

perform, using the trained prediction model, a subsequent update of the updated optimal per-stage task execution parallelism level of each respective stage of the plurality of stages of the distributed application based on the historic metrics collected during the at least one of the subsequent runs of the distributed application and the runs of the set of similar distributed applications, wherein the computer system determines similarity between the distributed application and the set of similar distributed applications by comparing one or more of anonymized execution graphs, data dependencies, and binaries corresponding to the distributed application and each of the set of similar distributed applications.

8 . The computer system of claim 7 , wherein the processor further executes the program instructions to:

run a set of artificial workloads on the distributed application;

generate generic metrics corresponding to the execution of the set of tasks in each respective stage of the plurality of stages of the distributed application based on running the artificial workloads on the distributed application; and

generate a fingerprint for each respective stage of the plurality of stages of the distributed application based on the generic metrics corresponding to the execution of the set of tasks in each respective stage of the plurality of stages of the distributed application.

9 . The computer system of claim 8 , wherein the processor further executes the program instructions to:

train a prediction model initially offline to form the trained prediction model using the fingerprint of each respective stage of the plurality of stages of the distributed application that was based on the generic metrics corresponding to the execution of the set of tasks in each respective stage of the plurality of stages; and

determine, using the trained prediction model, the optimal per-stage task execution parallelism level of each respective stage of the plurality of stages of the distributed application for the first run of the distributed application.

10 . The computer system of claim 9 , wherein the processor further executes the program instructions to:

schedule the set of tasks of each respective stage of the plurality of stages of the distributed application to execute on the set of worker nodes during the first run of the distributed application based on the optimal per-stage task execution parallelism level determined for each respective stage of the plurality of stages; and

collect the metrics corresponding to the execution of the set of tasks in each respective stage of the plurality of stages of the distributed application from the set of worker nodes during the first run of the distributed application.

11 . A computer program product for scheduling tasks for concurrent execution in parallel, the computer program product comprising a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:

receive, from a set of worker nodes via a network, metrics corresponding to execution of tasks in a plurality of stages of a distributed application that are running on the set of worker nodes;

perform, using a trained prediction model, an analysis of the metrics corresponding to the execution of the tasks in the plurality of stages of the distributed application received from the set of worker nodes;

determine, using the trained prediction model, an optimal per-stage task execution parallelism level for each respective stage of the plurality of stages based on the analysis of the metrics corresponding to the execution of the tasks in the plurality of stages of the distributed application;

schedule the tasks to execute on the set of worker nodes based on the optimal per-stage task execution parallelism level determined for each respective stage of the plurality of stages to improve at least one of runtime of the distributed application and resource utilization by the distributed application;

update, using the trained prediction model, the optimal per-stage task execution parallelism level of each respective stage of the plurality of stages of the distributed application after a first run of the distributed application to form an updated optimal per-stage task execution parallelism level based on the metrics corresponding to execution of a set of tasks in each respective stage of the plurality of stages of the distributed application collected from the set of worker nodes during the first run of the distributed application;

utilize the updated optimal per-stage task execution parallelism level to schedule the set of tasks in each respective stage of the plurality of stages of the distributed application on the set of worker nodes during subsequent runs of the distributed application;

collect historic metrics during at least one of the subsequent runs of the distributed application and runs of a set of similar distributed applications; and

perform, using the trained prediction model, a subsequent update of the updated optimal per-stage task execution parallelism level of each respective stage of the plurality of stages of the distributed application based on the historic metrics collected during the at least one of the subsequent runs of the distributed application and the runs of the set of similar distributed applications, wherein the computer determines similarity between the distributed application and the set of similar distributed applications by comparing one or more of anonymized execution graphs, data dependencies, and binaries corresponding to the distributed application and each of the set of similar distributed applications.

12 . The computer program product of claim 11 , wherein the program instructions further cause the computer to:

run a set of artificial workloads on the distributed application;

generate generic metrics corresponding to the execution of the set of tasks in each respective stage of the plurality of stages of the distributed application based on running the artificial workloads on the distributed application; and

generate a fingerprint for each respective stage of the plurality of stages of the distributed application based on the generic metrics corresponding to the execution of the set of tasks in each respective stage of the plurality of stages of the distributed application.

13 . The computer program product of claim 12 , wherein the program instructions further cause the computer to:

train a prediction model initially offline to form the trained prediction model using the fingerprint of each respective stage of the plurality of stages of the distributed application that was based on the generic metrics corresponding to the execution of the set of tasks in each respective stage of the plurality of stages; and

determine, using the trained prediction model, the optimal per-stage task execution parallelism level of each respective stage of the plurality of stages of the distributed application for the first run of the distributed application.

14 . The computer program product of claim 13 , wherein the program instructions further cause the computer to:

schedule the set of tasks of each respective stage of the plurality of stages of the distributed application to execute on the set of worker nodes during the first run of the distributed application based on the optimal per-stage task execution parallelism level determined for each respective stage of the plurality of stages; and

collect the metrics corresponding to the execution of the set of tasks in each respective stage of the plurality of stages of the distributed application from the set of worker nodes during the first run of the distributed application.