Multi-level action lookup table for dynamic throttling adjustments
A software-based activity monitor may dynamically change throttling criteria applied by a hardware-based activity monitor to a processing engine without interruption. The hardware-based activity monitory may use a multi-level activity table to determine throttling criteria to apply. The software-based activity monitor can rewrite a remapping sub-table of the multi-level action table in a single write in order to change which throttling criteria are determined as part of accessing the multi-level action table.
1 . A system, comprising:
at least one processor;
a memory, storing program instructions that when executed by the at least one processor cause the at least one processor to submit one or more machine learning tasks for processing to an inference accelerator; and
wherein the inference accelerator comprises an integrated circuit, wherein the integrated circuit comprises a software-based activity monitor implemented by firmware executing on a first portion of the integrated circuit, wherein the integrated circuit further comprises a tensor processing unit that comprises a hardware-based activity monitor implemented on a second portion of the integrated circuit and a systolic array implemented on a third portion of the integrated circuit that is configured to perform the one or more machine learning tasks; and
wherein the software-based activity monitor is configured to:
determine an update to throttling criteria to be applied by the hardware-based activity monitor; and
while the hardware-based activity monitor is monitoring activity of the systolic array:
perform a write to modify a remapping sub-table of a multi-level action table for the hardware-based activity monitor to point to a different entry in a throttling sub-table of the multiple level action table, wherein different sub-tables of the multi-level action table, including the remapping sub-table, are accessed to determine the throttling criteria used by the hardware-based activity monitor to evaluate activity of the systolic array and to modify, according to the evaluation, a rate at which data enters the systolic array for processing.
2 . The system of claim 1 , wherein as part of evaluating the activity of the systolic array, the hardware-based activity monitor is configured to:
determine an activity sub-table entry of the multilevel action table according to a measurement value for a measurement period;
access the remapping sub-table entry of the multi-level action table according to the determined activity sub-table entry; and
access a throttling sub-table entry of the multi-level action table according to the determined remapping sub-table entry to determine a particular throttling criteria to consider.
3 . The system of claim 2 , wherein to determine the activity sub-table entry of the multilevel action table according to the measurement value for the measurement period, the hardware-based activity monitor is configured to compare the measurement value with a hysteresis high field to determine that a hysteresis bit, used to determine the activity sub-table entry, is set.
4 . The system of claim 2 , wherein to determine the activity sub-table entry of the multilevel action table according to the measurement value for the measurement period, the hardware-based activity monitor is configured to compare the measurement value with a hysteresis low field to determine that a hysteresis bit, used to determine the activity sub-table entry, is not set.
5 . A method, comprising:
determining, by a software-based activity monitor implemented according to program instructions executing on a first portion of an integrated circuit, an update to throttling criteria to be applied by a hardware-based activity monitor implemented on a second portion of the integrated circuit; and
while the hardware-based activity monitor is monitoring activity of a third portion of the integrated circuit:
performing, by the software-based activity monitor, a write to modify a remapping sub-table of a multi-level action table for the hardware-based activity monitor to point to a different entry in a throttling sub-table of the multiple level action table, wherein different sub-tables of the multi-level action table, including the remapping sub-table, are accessed to determine the throttling criteria used by the hardware-based activity monitor to evaluate activity of the third portion of the integrated circuit and to modify, according to the evaluation, a rate at which data enters the third portion of the integrated circuit for processing.
6 . The method of claim 5 , further comprising:
as part of evaluating the activity of the third portion of the integrated circuit:
determining, by the hardware-based activity monitor, an activity sub-table entry of the multilevel action table according to a measurement value for a measurement period;
accessing, by the hardware-based activity monitor, the remapping sub-table entry of the multi-level action table according to the determined activity sub-table entry; and
accessing, by the hardware-based activity monitor, a throttling sub-table entry of the multi-level action table according to the determined remapping sub-table entry to determine a particular throttling criteria to consider.
7 . The method of claim 6 , wherein determining the activity sub-table entry of the multilevel action table according to the measurement value for the measurement period comprises comparing the measurement value with a hysteresis low field to determine that a hysteresis bit, used to determine the activity sub-table entry, is not set.
8 . The method of claim 6 , wherein determining the activity sub-table entry of the multilevel action table according to the measurement value for the measurement period comprises comparing the measurement value with a hysteresis high field to determine that a hysteresis bit value, used to determine the activity sub-table entry, is set.
9 . The method of claim 5 , further comprising:
determining, by the software-based activity monitor, a second update to second throttling criteria to be applied by a second hardware-based activity monitor implemented on a fourth portion of the integrated circuit; and
while the second hardware-based activity monitor is monitoring activity of a fifth portion of the integrated circuit:
performing, by the software-based activity monitor, a write to modify a second remapping sub-table of a second multi-level action table for the second hardware-based activity monitor;
wherein the update to the second throttling criteria causes the second throttling criteria to be different than the updated throttling criteria.
10 . The method of claim 5 , wherein the processing performed on the second portion of the integrated circuit is performed on a systolic array.
11 . The method of claim 10 , wherein the multi-level action table is implemented as part of an array utilization monitor in the hardware-based activity monitor.
12 . The method of claim 10 , wherein the multi-level action table is implemented as part of an array data activity monitor in the hardware-based activity monitor.
13 . A device, comprising circuitry that implements an integrated circuit, the integrated circuit comprising:
a first portion that implements a processing engine;
a second portion that implements a hardware-based activity monitor;
a third portion executing programing instructions that implement a software-based activity monitor, the software-based activity monitor configured to:
determine an update to throttling criteria to be applied by the hardware-based activity monitor; and
while the hardware-based activity monitor is monitoring activity of the processing engine:
perform a write to modify a remapping sub-table of a multi-level action table for the hardware-based activity monitor to point to a different entry in the a throttling sub-table of the multiple level action table, wherein different sub-tables of the multi-level action table, including the remapping sub-table, are accessed to determine the throttling criteria used by the hardware-based activity monitor to evaluate activity of the processing engine and to modify, according to the evaluation, a rate at which data enters the processing engine for processing.
14 . The device of claim 13 , wherein as part of evaluating the activity of the first portion of the integrated circuit, the hardware-based activity monitor is configured to:
determine an activity sub-table entry of the multilevel action table according to a measurement value for a measurement period;
access the remapping sub-table entry of the multi-level action table according to the determined activity sub-table entry; and
access a throttling sub-table entry of the multi-level action table according to the determined remapping sub-table entry to determine a particular throttling criteria to consider.
15 . The device of claim 14 , wherein to determine the activity sub-table entry of the multilevel action table according to the measurement value for the measurement period, the hardware-based activity monitor is configured to compare the measurement value with a hysteresis low field to determine that a hysteresis bit, used to determine the activity sub-table entry, is not set.
16 . The device of claim 14 , wherein to determine the activity sub-table entry of the multilevel action table according to the measurement value for the measurement period, the hardware-based activity monitor is configured to compare the measurement value with a hysteresis high field to determine that a hysteresis bit, used to determine the activity sub-table entry, is set.
17 . The device of claim 14 , wherein the software-based activity monitor is further configured to:
determine a second update to second throttling criteria to be applied by a different component of the hardware-based activity monitor; and
while the hardware-based activity monitor is monitoring activity of the processing engine, perform a write to modify a second remapping sub-table of the multi-level action table for the hardware-based activity monitor.
18 . The device of claim 13 , wherein the processing engine is a systolic array and wherein the multi-level action table is implemented as part of an array utilization monitor in the hardware-based activity monitor.
19 . The device of claim 13 , wherein the processing engine is a systolic array and wherein the multi-level action table is implemented as part of an array data activity monitor in the hardware-based activity monitor.
20 . The device of claim 14 , wherein the integrated circuit is an application specific integrated circuit (ASIC), field programmable gate array (FPGA), or system-on-a-chip (SoC).