Resolving cluster computing task interference
A system for resolving interference on a cluster computing system includes a processor. The processor is configured to determine whether a second worker system has resources available to run the first or second task; in response to determining that the second worker system has the resources available to run the second task, move the second task from the first to the second worker system; in response to determining that the second worker system has the resources available to run the first task and does not have resources available to run the second task, move the first task from the first to the second worker system; and in response to determining that the second worker system does not have the resources available to run the first or second task, limit a resource allocation of the first or second task.
1. A system for resolving interference on a cluster computing system, comprising:
an input interface configured to:
receive an indication of a task interference on a first worker system of a plurality of worker systems, wherein the task interference comprises a reduction in performance of a first task when a second task is running concurrently on the first worker system; and
a processor configured to:
determine whether a second worker system of the plurality of worker systems has resources available to run the first task or the second task based on a first priority associated with the first task and a second priority associated with the second task;
in response to determining that the second worker system has the resources available to run the second task:
move the second task from the first worker system to the second worker system;
in response to determining that the second worker system has the resources in response to determining that the second worker system has the resources available to run the second task:
move the first task from the first worker system to the second worker system;
in response to determining that the second worker system does not have the resources available to run the first task or the second task:
limit a resource allocation of the first task or the second task; and
move all tasks of a task group, wherein the task group comprises a subset of a set of tasks comprising multiple instances of the same task.
2. The system of claim 1 , wherein the indication of task interference comprises an indication of a sudden reduction in a resource performance.
3. The system of claim 1 , wherein the task interference comprises an indication of a sudden reduction in performance of one or more key tasks.
4. The system of claim 1 , wherein an indication of a task interference comprises an indication that a combined resource utilization of a first task and a second task both running on the first worker system is greater than a resource capacity of the first worker system.
5. The system of claim 1 , wherein limiting the resource allocation of the first task or the second task comprises limiting the resource allocation of the first task and/or the second task.
6. The system of claim 1 , wherein a task resource allocation is unlimited by default.
7. The system of claim 1 , wherein a resource of the resources comprises a network bandwidth, a disk bandwidth, a memory bandwidth, or a processor cache.
8. The system of claim 1 , wherein the second worker system comprises a most loaded worker system of the plurality of worker systems that has the resources available to run the second task.
9. The system of claim 1 , wherein the second worker system comprises a most loaded worker system of the plurality of worker systems that has the resources available to run the first task.
10. The system of claim 1 , wherein determining that the second worker system does not have the resources available to run the first task or the second task comprises determining that no worker system of the plurality of worker systems has resources available to run the first task or the second task.
11. The system of claim 1 , wherein the processor is additionally configured to add the indication of task interference to a task interference database.
12. The system of claim 11 , wherein the task interference database comprises information describing a plurality of task interference events.
13. The system of claim 12 , wherein the task interference database is used by the computing cluster for allocating tasks to worker systems of the plurality of worker systems with reduced interference.
14. The system of claim 1 , wherein the processor is further configured to receive an indication of a performance target for the first task.
15. The system of claim 14 , wherein limiting a resource is based at least in part on the performance target.
16. The system of claim 15 , wherein the processor is further configured to, in the event a limit is below a requirement for the performance target, allocate a new cluster machine for the first task.
17. A method for resolving interference on a cluster computing system, comprising:
receiving an indication of a task interference on a first worker system of a plurality of worker systems, wherein the task interference comprises a reduction in performance of a first task when a second task is running concurrently on the first worker system;
determining, using a processor, whether a second worker system of the plurality of worker systems has resources available to run the first task or the second task based on a first priority associated with the first task and a second priority associated with the second task;
in response to determining that the second worker system has the resources available to run the second task:
moving the second task from the first worker system to the second worker system;
in response to determining that the second worker system has the resources available to run the first task and does not have resources available to run the second task:
moving the first task from the first worker system to the second worker system;
in response to determining that the second worker system does not have the resources available to run the first task or the second task:
limiting a resource allocation of the first task or the second task; and
moving all tasks of a task group, wherein the task group comprises a subset of a set of tasks comprising multiple instances of the same task.
18. A computer program product for resolving interference on a cluster computing system, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving an indication of a task interference on a cluster machine of a plurality of cluster machines comprising a computer cluster running a set of tasks;
receiving an indication of a task interference on a first worker system of a plurality of worker systems, wherein the task interference comprises a reduction in performance of a first task when a second task is running concurrently on the first worker system;
determining whether a second worker system of the plurality of worker systems has resources available to run the first task or the second task based on a first priority associated with the first task and a second priority associated with the second task;
in response to determining that the second worker system has the resources available to run the second task:
moving the second task from the first worker system to the second worker system;
in response to determining that the second worker system has the resources available to run the second task:
moving the first task from the first worker system to the second worker system;
in response to determining that the second worker system does not have the resources available to run the first task or the second task:
limiting a resource allocation of the first task or the second task; and
moving all tasks of a task group, wherein the task group comprises a subset of a set of tasks comprising multiple instances of the same task.