Proactive load-balancing using retroactive work refusal

1. A method by a computing system implementing a cluster of nodes to load balance among the cluster of nodes using retroactive refusal, each of the nodes within the cluster of nodes capable of processing work items, the method comprising:

accepting, by a first node within the cluster of nodes, a work item to be processed by the first node;

processing the work item on the first node, wherein the work item is a discrete computing task and processing of the work item involves executing computer code to process dynamically generated data pertaining to the discrete computing task as the data is being generated;

during the processing of the work item on the first node, monitoring a resource utilization of the first node;

determining whether the resource utilization of the first node exceeds a predetermined threshold;

determining that the work item is to be retroactively refused by the first node in response to a determination that the resource utilization of the first node exceeds the predetermined threshold;

setting, by the first node, a priority of the work item to a first priority that is higher than a current priority of the work item in response to the determination that the work item is to be retroactively refused by the first node, wherein the current priority of the work item is a priority with which the work item was accepted by the first node for processing and the work item is to be reassigned to another node within the cluster of nodes for processing using the first priority after the work item is retroactively refused by the first node;

subsequent to the first node setting the priority of the work item to the first priority, retroactively refusing, by the first node, the work item, wherein the retroactive refusing by the first node includes:

terminating the work item;

marking the work item as being unassigned; and

moving the work item to a pool of unassigned work items to be picked up and accepted by another node within the cluster of nodes using the first priority;

picking up and accepting, by a second node within the cluster of nodes, the work item from the pool of unassigned work items ahead of other unassigned work items in the pool of unassigned work items having a lower priority than the first priority;

subsequent to the second node picking up and accepting the work item, processing the work item on the second node, which involves executing computer code;

during the processing of the work item on the second node, setting, by the second node, the priority of the work item to a second priority that is higher than the first priority, in response to a determination that the work item is to be retroactively refused by the second node; and

subsequent to the second node setting the priority of the work item to the second priority, retroactively refusing, by the second node, the work item, wherein the retroactive refusing by the second node includes:

terminating the work item;

marking the work item as being unassigned; and

moving the work item to the pool of unassigned work items.

2. The method of claim 1 , wherein the resource utilization includes any one of: a central processing unit (CPU) utilization, an input/output (I/O) utilization, and a network bandwidth utilization.

3. The method of claim 1 , wherein the work item includes a video processing task or a log processing task.

4. The method of claim 1 , wherein the work item is assigned to the first node by a load balancer.

5. A computing system configured to implement a cluster of nodes that provides load balancing among the cluster of nodes using retroactive refusal, each of the nodes within the cluster of nodes capable of processing work items, the computing system comprising:

one or more processors; and

a non-transitory machine-readable storage medium having instructions stored therein, which when executed by the one or more processors, causes the computing system to:

accept, by a first node within the cluster of nodes, a work item to be processed by the first node;

process the work item on the first node, wherein the work item is a discrete computing task and the processing of the work item involves a further step to execute computer code to process dynamically generated data pertaining to the discrete computing task as the data is being generated;

during the processing of the work item on the first node, monitor a resource utilization of the first node;

determine whether the resource utilization of the first node exceeds a predetermined threshold;

determine that the work item is to be retroactively refused by the first node in response to a determination that the resource utilization of the first node exceeds the predetermined threshold;

set, by the first node, a priority of the work item to a first priority that is higher than a current priority of the work item in response to the determination that the work item is to be retroactively refused by the first node, wherein the current priority of the work item is a priority with which the work item was accepted by the first node for processing and the work item is to be reassigned to another node within the cluster of nodes for processing using the first priority after the work item is retroactively refused by the first node;

subsequent to the first node setting the priority of the work item to the first priority, retroactively refuse, by the first node, the work item, wherein the retroactive refusing by the first node includes:

terminate the work item;

mark the work item as being unassigned; and

move the work item to a pool of unassigned work items to be picked up and accepted by another node within the cluster of nodes using the first priority;

pick up and accept, by a second node within the cluster of nodes, the work item from the pool of unassigned work items ahead of other unassigned work items in the pool of unassigned work items having a lower priority than the first priority;

subsequent to the second node picking up and accepting the work item, process the work item on the second node, which involves executing computer code;

during the processing of the work item on the second node, set, by the second node, the priority of the work item to a second priority that is higher than the first priority, in response to a determination that the work item is to be retroactively refused by the second node; and

subsequent to the second node setting the priority of the work item to the second priority, retroactively refuse, by the second node, the work item, wherein the retroactive refusing by the second node includes:

terminate the work item;

mark the work item as being unassigned; and

move the work item to the pool of unassigned work items.

6. The computing system of claim 5 , wherein the resource utilization includes any one of: a central processing unit (CPU) utilization, an input/output (I/O) utilization, and a network bandwidth utilization.

7. The computing system of claim 5 , wherein the work item is assigned to the first node by a load balancer.

8. A non-transitory machine-readable storage medium having instructions stored therein, which when executed by one or more processors of a computing system implementing a cluster of nodes, causes the computing system to perform operations for providing load balancing among the cluster of nodes using retroactive refusal, each of the nodes within the cluster of nodes capable of processing work items, the operations comprising:

accepting, by a first node within the cluster of nodes, a work item to be processed by the first node;

processing the work item on the first node, wherein the work item is a discrete computing task and processing of the work item involves executing computer code to process dynamically generated data pertaining to the discrete computing task as the data is being generated;

during the processing of the work item on the first node, monitoring a resource utilization of the first node;

determining whether the resource utilization of the first node exceeds a predetermined threshold;

determining that the work item is to be retroactively refused by the first node in response to a determination that the resource utilization of the first node exceeds the predetermined threshold;

setting, by the first node, a priority of the work item to a first priority that is higher than a current priority of the work item in response to the determination that the work item is to be retroactively refused by the first node, wherein the current priority of the work item is a priority with which the work item was accepted by the first node for processing and the work item is to be reassigned to another node within the cluster of nodes for processing using the first priority after the work item is retroactively refused by the first node;

subsequent to the first node setting the priority of the work item to the first priority, retroactively refusing, by the first node, the work item, wherein the retroactive refusing by the first node includes:

terminating the work item;

marking the work item as being unassigned; and

moving the work item to a pool of unassigned work items to be picked up and accepted by another node within the cluster of nodes using the first priority;

picking up and accepting, by a second node within the cluster of nodes, the work item from the pool of unassigned work items ahead of other unassigned work items in the pool of unassigned work items having a lower priority than the first priority;

subsequent to the second node picking up and accepting the second node, processing the work item on the second node, which involves executing computer code;

during the processing of the work item on the work item, setting, by the second node, the priority of the work item to a second priority that is higher than the first priority, in response to a determination that the work item is to be retroactively refused by the second node; and

subsequent to the second node setting the priority of the work item to the second priority, retroactively refusing, by the second node, the work item, wherein the retroactive refusing by the second node includes:

terminating the work item;

marking the work item as being unassigned; and

moving the work item to the pool of unassigned work items.

9. The non-transitory machine-readable storage medium of claim 8 , wherein the resource utilization includes any one of: a central processing unit (CPU) utilization, an input/output (I/O) utilization, and a network bandwidth utilization.

10. The non-transitory machine-readable storage medium of claim 8 , wherein the work item is assigned to the first node by a load balancer.