System and method for data application performance management

1. A computer-implemented method, comprising:

receiving a selection of a goal for an application on a cluster of compute nodes, wherein the cluster of compute nodes is part of a big data system and wherein the goal includes one or more of a speedup goal, an efficiency goal, a reliability goal, and a service level agreement goal;

collecting predicted values of metrics associated with the goal, the big data system, and the application using cost-based and rule-based query optimization techniques and machine learning;

executing the application on the cluster of compute nodes and collecting data associated with the goal from the big data system;

determining, based on the data and predicted values, a recommendation to adjust one or more parameters that would allow the goal to be achieved to optimize running the application on the cluster of compute nodes of the big data system; and

providing the recommendation to adjust the one or more parameters.

2. The computer-implemented method of claim 1 , further comprising providing an explanation of the recommendation.

3. The computer-implemented method of claim 1 , wherein determining the recommendation further comprises analyzing data from a data source, the data source including one or more of (a) data collected in a current execution session; (b) data collected from previous execution sessions of the application; and (c) data related to a past use of the recommendation.

4. The computer-implemented method of claim 3 , further comprising:

collecting observed metrics by running the application with a set of recommendations; and

collecting predicted values of metrics without running the application.

5. The computer-implemented method of claim 3 , further comprising running the application on datasets that are specific to a user.

6. The computer-implemented method of claim 3 , further comprising executing the application in a specific resource pool.

7. The computer-implemented method of claim 1 , further comprising modifying the parameters while executing the application on the cluster.

8. The computer-implemented method of claim 1 , further comprising modifying the goal while executing the application on the cluster.

9. The computer-implemented method of claim 1 , wherein the application is for one of MapReduce, Spark, Impala, Hive, Tez, LLAP, Kafka, and SQL.

10. The computer implemented method of claim 1 , wherein the data collected includes observed metrics obtained by running the application with a set of recommendations.

11. The computer-implemented method of claim 1 , wherein collecting data associated with the goal includes using a probe that determines a configuration that meets the goal.

12. The computer-implemented method of claim 11 , wherein determining the recommendation further comprises executing the application using the parameter over multiple iterations; analyzing historical data; and analyzing probe data generated using the configuration.

13. The computer-implemented method of claim 11 , wherein the probe implements one or more models, the models including true, proxy, rule, probabilistic, and hybrid.

14. The computer-implemented method of claim 1 , wherein the recommendation is a set of recommendations.

15. The computer-implemented method of claim 1 , wherein the parameters include one or more of maximum availability parallelism, container size, and a number of partitions of the cluster.

16. The computer-implemented method of claim 1 , further comprising displaying a user interface, the user interface including one or more user selectable actions.

17. The computer-implemented method of claim 16 , wherein the one or more user selectable actions include configure, compare, autotune, analyze, apply, enforce, and watch.

18. The computer-implemented method of claim 1 , further comprising transmitting the recommendation to a distributed storage system of the big data system to increase the performance of the distributed storage system.

19. A computer-implemented method, comprising:

receiving a selection of a goal for an application on a cluster of compute nodes, wherein the cluster of compute nodes are part of a big data system and wherein the goal includes one or more of a speedup goal, an efficiency goal, a reliability goal, and a service level agreement goal;

collecting predicted values of metrics associated with the goal, the big data system, and the application using cost-based and rule-based query optimization techniques and machine learning;

determining, based on the predicted values, a recommendation to adjust one or more parameters that would allow the goal to be achieved to optimize running the application on the cluster of compute nodes of the big data system; and

providing the recommendation to adjust the one or more parameters.

20. The computer-implemented method of claim 19 , wherein determining the recommendation further comprises analyzing data from a data source, the data source including one or more of (a) data collected in a current execution session; (b) data collected from previous execution sessions of the application; and (c) data related to a past use of the recommendation.

21. The computer-implemented method of claim 20 , further comprising:

collecting observed metrics by running the application with a set of recommendations; and

collecting predicted values of metrics without running the application by using cost-based query optimizers.