Task dispatch monitoring for dynamic adaptation to system conditions
Monitoring system wide task dispatch allows dynamic adaptation to conditions of a system. A monitor tracks the total tasks currently dispatched to the system. In a system with multiple processing units, this monitor is centralized and collects information about tasks dispatched to each of the processing units. The monitor compares the total dispatched tasks against a threshold that has already been defined. Further dispatching of tasks to the system is regulated based on comparison of the total dispatched tasks against the threshold. If the comparison achieves a trigger condition (e.g., total dispatched tasks exceeding the threshold), then task dispatch is throttled. Throttling further task dispatching, as long as the threshold is exceeded, allows progress to continue without overwhelming the system.
1. A method comprising:
receiving, on a processor, task information from a plurality of processing units of a system, a first portion of the task information being received from a first processing unit of the plurality of processing units and identifying a first number of queued tasks on the first processing unit, a second portion of the task information being received from a second processing unit of the plurality of processing units and identifying a second number of queued tasks on the second processing unit;
generating a plurality of compile tasks at a first task dispatcher, wherein the first number of queued tasks on the first processing unit includes at least a portion of the plurality of compile tasks;
determining, on the processor, system wide task information based on receipt of the task information, the system wide task information including a total number of system wide queued tasks in the system, the first number of queued tasks included among the total number of system wide queued tasks from the first task dispatcher and the second number of queued tasks included among the total number of system wide queued tasks from a second task dispatcher;
determining, on the processor, a condition of the system based on a comparison of the total number of system wide queued tasks to a system task threshold, the system task threshold being based on a resource utilization condition and an overload condition associated with the plurality of processing units; and
throttling, on the processor, task dispatch from the first task dispatcher and the second task dispatcher based on a result of a determination of the condition.
2. The method of claim 1 , wherein throttling the task dispatch comprises:
setting a task dispatch flag associated with the first task dispatcher and the second task dispatcher based on the result of the comparison.
3. The method of claim 1 , further comprising:
examining runtime feedback of code generated from a plurality of previously executed compile commands; and
generating a compile command from a plurality of previous compiler options based on examination of the runtime feedback,
wherein generation of the plurality of compile tasks is based on the compile command.
4. The method of claim 1 , wherein throttling the task dispatch comprises:
reducing the task dispatch from the first task dispatcher and the second task dispatcher to allow progress of currently dispatched tasks and avoid oversubscription of the system.
5. The method of claim 1 , wherein the system task threshold comprises a multiple of a number of the plurality of processing units.
6. The method of claim 1 , wherein determining the condition comprises:
determining the condition of the system based on the comparison of the total number of system wide queued tasks to a multiple of a number of the plurality of processing units and utilization of the plurality of processing units.
7. A non-transitory machine-readable medium comprising instructions, which when executed by a processor, cause the processor to:
receive task information from a plurality of processing units of a system, a first portion of the task information being received from a first processing unit of the plurality of processing units and identifying a first number of queued tasks on the first processing unit, a second portion of the task information being received from a second processing unit of the plurality of processing units and identifying a second number of queued tasks on the second processing unit;
generate a plurality of compile tasks at a first task dispatcher, wherein the number of queued tasks on the first processing unit includes at least a portion of the plurality of compile tasks;
determine system wide task information based on receipt of the task information, the system wide task information including a total number of system wide queued tasks in the system, the first number of queued tasks included among the total number of system wide queued tasks from the first task dispatcher and the second number of queued tasks included among the total number of system wide queued tasks from a second task dispatcher;
determine a condition of the system based on a comparison of the total number of system wide queued tasks to a system task threshold, the system task threshold being based on a resource utilization condition and an overload condition associated with the plurality of processing units; and
throttle task dispatch the first task dispatcher and the second task dispatcher based on a result of a determination of the condition.
8. The non-transitory machine-readable medium of claim 7 , wherein the system task threshold comprises a multiple of a number of the plurality of processing units.
9. The non-transitory machine-readable medium of claim 7 , wherein the instructions that cause the processor to determine the condition causes the processor to:
determine the condition of the system based on the comparison of the total number of system wide queued tasks to a multiple of a number of the plurality of processing units and utilization of the plurality of processing units.
10. The non-transitory machine-readable medium of claim 7 , wherein the instructions that cause the processor to throttle the task dispatch causes the processor to:
set a task dispatch flag associated with the first task dispatcher and the second task dispatcher based on the result of the comparison.
11. The non-transitory machine-readable medium of claim 7 , wherein the instructions cause the processor to:
examine runtime feedback of code generated from a plurality of previously executed compile commands; and
generate a compile command from a plurality of previous compiler options based on examination of the runtime feedback,
wherein generation of the plurality of compile tasks is based on the compile command.
12. A system comprising:
a processor and a memory coupled to the processor; and
a module deployed in the memory and executed by the processor to:
receive a first portion of task information from a plurality of processing units of a system, the task information being received from a first processing unit of the plurality of processing units and identifying a first number of queued tasks on the first processing unit, a second portion of the task information being received from a second processing unit of the plurality of processing units and identifying a second number of queued tasks on the second processing unit;
generate a plurality of compile tasks at a first task dispatcher, wherein the number of queued tasks on the first processing unit includes at least a portion of the plurality of compile tasks;
determine system wide task information based on receipt of the task information, the system wide task information including a total number of system wide queued tasks in the system, the first number of queued tasks included among the total number of system wide queued tasks from the first task dispatcher and the second number of queued tasks included among the total number of system wide queued tasks from a second task dispatcher;
determine a condition of the system based on a comparison of the total number of system wide queued tasks to a system task threshold, the system task threshold being based on a resource utilization condition and an overload condition associated with the plurality of processing units; and
throttle task dispatch the first task dispatcher and the second task dispatcher based on a result of a determination of the condition.
13. The system of claim 12 , wherein the module is configured to:
establish the system task threshold,
wherein throttling the task dispatch is based on establishment of the system task threshold.
14. The system of claim 12 , wherein the system task threshold comprises a multiple of a number of the plurality of processing units.
15. The system of claim 12 , wherein the module that causes the processor to determine the condition causes the processor to:
determine the condition of the system based on the comparison of the total number of system wide queued tasks to a multiple of a number of the plurality of processing units and utilization of the plurality of processing units.
16. The system of claim 12 , wherein the module that causes the processor to throttle the task dispatch causes the processor to:
set a task dispatch flag associated with the first task dispatcher and the second task dispatcher based on the result of the comparison.
17. The system of claim 12 , wherein the module causes the processor to:
examine runtime feedback of code generated from a plurality of previously executed compile commands; and
generate a compile command from a plurality of previous compiler options based on examination of the runtime feedback,
wherein generation of the plurality of compile tasks is based on the compile command.