Task-execution planning using machine learning

1. A method comprising:

receiving a task to be executed using resources of a computing cluster;

generating a task execution plan;

accessing information about data to be used for the task;

predicting, by at least one hardware processor, resource requirements for executing the task by applying machine learning to the task execution plan and the information about the data;

determining a total amount of time from submission of the task to termination of the task, the determining the total amount of the time comprising:

generating a prediction profile; and

managing assignment of a query based on resource data for the task; and

generating assignment data to execute the task on the resources by applying machine learning information about a current state of the resources and the predicted resource requirements.

2. The method of claim 1 , further comprising:

identifying an optimal number of tasks, the optimal number of tasks depending on a specific type of the task being executed; and

performing task coordination based on the optimal number of tasks, the task coordination including determining a number of tasks to execute on the computing cluster.

3. The method of claim 2 , wherein the identifying the optimal number of tasks further comprises:

identifying a characteristic of each task; and

identifying one or more requirements that each task imposes on the computing cluster.

4. The method of claim 1 , further comprising:

implementing, on a cloud data platform, improved task scheduling on a multi-server cluster for data-related workloads, the improved task scheduling including improving a throughput with minimal increase in a latency of the task.

5. The method of claim 1 , wherein each task comprises a query for execution on a dataset of the computing cluster.

6. A system comprising:

one or more hardware processors of a machine; and

at least one memory storing instructions that, when executed by the one or more hardware processors, cause the machine to perform operation comprising:

receiving a task to be executed using resources of a computing cluster;

generating a task execution plan;

accessing information about data to be used for the task;

predicting, by at least one hardware processor, resource requirements for executing the task by applying machine learning to the task execution plan and the information about the data;

determining a total amount of time from submission of the task to termination of the task, the determining the total amount of the time comprising:

generating a prediction profile; and

managing assignment of a query based on resource data for the task; and

generating assignment data to execute the task on the resources by applying machine learning information about a current state of the resources and the predicted resource requirements.

7. The system of claim 6 , the operations further comprising:

identifying an optimal number of tasks, the optimal number of tasks depending on a specific type of the task being executed; and

performing task coordination based on the optimal number of tasks, the task coordination including determining a number of tasks to execute on the computing cluster.

8. The system of claim 7 , wherein the identifying the optimal number of tasks further comprises:

identifying a characteristic of each task; and

identifying one or more requirements that each task imposes on the computing cluster.

9. The system of claim 8 , the operations further comprising:

implementing, on a cloud data platform, improved task scheduling on a multi-server cluster for data-related workloads, the improved task scheduling including improving a throughput with minimal increase in a latency of the task.

10. The system of claim 6 , wherein each task comprises a query for execution on a dataset of the computing cluster.

11. A machine-storage media embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:

receiving a task to be executed using resources of a computing cluster;

generating a task execution plan;

accessing information about data to be used for the task;

predicting, by at least one hardware processor, resource requirements for executing the task by applying machine learning to the task execution plan and the information about the data;

determining a total amount of time from submission of the task to termination of the task, the determining the total amount of the time comprising:

generating a prediction profile; and

managing assignment of a query based on resource data for the task; and

generating assignment data to execute the task on the resources by applying machine learning information about a current state of the resources and the predicted resource requirements.

12. The machine-storage media of claim 11 , the operations further comprising:

identifying an optimal number of tasks, the optimal number of tasks depending on a specific type of the task being executed; and

performing task coordination based on the optimal number of tasks, the task coordination including determining a number of tasks to execute on the computing cluster.

13. The machine-storage media of claim 12 , wherein the identifying the optimal number of tasks further comprises:

identifying a characteristic of each task; and

identifying one or more requirements that each task imposes on the computing cluster.

14. The machine-storage media of claim 13 , the operations further comprising:

implementing, on a cloud data platform, improved task scheduling on a multi-server cluster for data-related workloads, the improved task scheduling including improving a throughput with minimal increase in a latency of the task.