Variable power mode inferencing
Certain aspects of the present disclosure provide a method for performing multimode inferencing, comprising: receiving machine learning model input data from a requestor; processing the machine learning model input data with a machine learning model using processing hardware at a first power level to generate first output data; selecting a second power level for the processing hardware based on comparing the first output data to a threshold value; processing the machine learning model input data with the machine learning model using the processing hardware at the second power level to generate second output data; and sending second output data to the requestor.
1. A method for performing multimode inferencing, comprising:
receiving machine learning model input data from a requestor;
processing the machine learning model input data with a machine learning model using processing hardware at a first power level to generate first output data, wherein the processing hardware comprises a processor operable at one of a plurality of power levels including the first power level and a second power level different from the first power level, each power level corresponding to a different performance level for the machine learning model;
enabling data quantization by a pre-processor after selecting the first power level;
selecting the second power level for the processing hardware based on comparing the first output data to a threshold value;
disabling data quantization by the pre-processor after selecting the second power level;
processing the machine learning model input data with the machine learning model using the processing hardware at the second power level to generate second output data; and
sending second output data to the requestor.
2. The method of claim 1 , further comprising:
determining that a value of the first output data is less than the threshold value; and
determining that a value of the second output data is greater than or equal to the threshold value.
3. The method of claim 1 , wherein:
the first power level comprises a first voltage applied to the processing hardware,
the second power level comprises a second voltage applied to the processing hardware, and
the second voltage is higher than the first voltage.
4. The method of claim 1 , further comprising:
selecting the first power level for the processing hardware based on a first power mode associated with the machine learning model; and
selecting the second power level for the processing hardware based on a second power mode associated with the machine learning model.
5. The method of claim 1 , further comprising: quantizing the machine learning model input data according to a first power mode associated with the machine learning model.
6. The method of claim 1 , wherein:
the machine learning model comprises a linear classification model, and
comparing the first output data to a threshold value comprises comparing a magnitude of an output of the linear classification model to the threshold value η according to |f(x)|<η.
7. The method of claim 1 , wherein the processing hardware comprises an array of processing elements in an inference accelerator.
8. A method for performing multimode inferencing, comprising:
receiving machine learning model input data from a requestor;
processing the machine learning model input data with a machine learning model using processing hardware at a first power level to generate first output data, wherein the processing hardware comprises a processor operable at one of a plurality of power levels including the first power level and a second power level different from the first power level, each power level corresponding to a different performance level for the machine learning model;
selecting the second power level for the processing hardware based on comparing the first output data to a threshold value;
processing the machine learning model input data with the machine learning model using the processing hardware at the second power level to generate second output data; and
sending second output data to the requestor, wherein:
the machine learning model comprises a neural network model, and
comparing the first output data to a threshold value comprises comparing a maximum value of an activation function σ for the neural network model to the threshold value η according to: max σ(z) i max >η, where i max is an index for a final layer of the neural network model.
9. A method for performing multimode inferencing, comprising:
receiving machine learning model input data from a requestor;
processing the machine learning model input data with a machine learning model using processing hardware at a first power level to generate first output data, wherein the processing hardware comprises a processor operable at one of a plurality of power levels including the first power level and a second power level different from the first power level, each power level corresponding to a different performance level for the machine learning model;
selecting the second power level for the processing hardware based on comparing the first output data to a threshold value;
processing the machine learning model input data with the machine learning model using the processing hardware at the second power level to generate second output data; and
sending second output data to the requestor, wherein:
the machine learning model comprises a neural network model, and
comparing the first output data to a threshold value comprises comparing a difference of Softmax outputs for a first maximum candidate and a second maximum candidate to the threshold value.
10. A method for performing multimode inferencing, comprising:
receiving machine learning model input data from a requestor;
processing the machine learning model input data with a machine learning model using processing hardware at a first power level to generate first output data, wherein the processing hardware comprises a processor operable at one of a plurality of power levels including the first power level and a second power level different from the first power level, each power level corresponding to a different performance level for the machine learning model;
selecting the second power level for the processing hardware based on comparing the first output data to a threshold value;
processing the machine learning model input data with the machine learning model using the processing hardware at the second power level to generate second output data; and
sending second output data to the requestor, further comprising:
incrementing a counter value after selecting the second power level for the processing hardware; and
increasing the threshold value based on comparing the counter value to a counter threshold value.
11. A processing system for performing multimode inferencing, comprising:
a memory comprising computer-executable instructions;
one or more processors configured to execute the computer-executable instructions and cause the processing system to:
receive machine learning model input data from a requestor;
process the machine learning model input data with a machine learning model using processing hardware at a first power level to generate first output data, wherein the processing hardware comprises a processor operable at one of a plurality of power levels including the first power level and a second power level different from the first power level, each power level corresponding to a different performance level for the machine learning model;
enable data quantization by a pre-processor after selecting the first power level;
select the second power level for the processing hardware based on comparing the first output data to a threshold value;
disable data quantization by the pre-processor after selecting the second power level;
process the machine learning model input data with the machine learning model using the processing hardware at the second power level to generate second output data; and
send second output data to the requestor.
12. The processing system of claim 11 , wherein the one or more processors are further configured to cause the processing system to:
determine that a value of the first output data is less than the threshold value; and
determine that a value of the second output data is greater than or equal to the threshold value.
13. The processing system of claim 11 , wherein:
the first power level comprises a first voltage applied to the processing hardware,
the second power level comprises a second voltage applied to the processing hardware, and
the second voltage is higher than the first voltage.
14. The processing system of claim 11 , wherein the one or more processors are further configured to cause the processing system to:
select the first power level for the processing hardware based on a first power mode associated with the machine learning model; and
select the second power level for the processing hardware based on a second power mode associated with the machine learning model.
15. The processing system of claim 11 , wherein the one or more processors are further configured to cause the processing system to: quantizing the machine learning model input data according to a first power mode associated with the machine learning model.
16. The processing system of claim 11 , wherein:
the machine learning model comprises a linear classification model, and
in order to compare the first output data to a threshold value, the one or more processors are further configured to cause the processing system to compare a magnitude of an output of the linear classification model to the threshold value η according to |f(x)|<η.
17. A processing system for performing multimode inferencing, comprising:
a memory comprising computer-executable instructions;
one or more processors configured to execute the computer-executable instructions and cause the processing system to:
receive machine learning model input data from a requestor;
process the machine learning model input data with a machine learning model using processing hardware at a first power level to generate first output data, wherein the processing hardware comprises a processor operable at one of a plurality of power levels including the first power level and a second power level different from the first power level, each power level corresponding to a different performance level for the machine learning model;
select the second power level for the processing hardware based on comparing the first output data to a threshold value;
process the machine learning model input data with the machine learning model using the processing hardware at the second power level to generate second output data; and
send second output data to the requestor, wherein:
the machine learning model comprises a neural network model, and
in order to compare the first output data to a threshold value, the one or more processors are further configured to cause the processing system to compare a maximum value of an activation function σ for the neural network model to the threshold value η according to: max σ(z) i max >η, where i max is an index for a final layer of the neural network model.
18. A processing system for performing multimode inferencing, comprising:
a memory comprising computer-executable instructions;
one or more processors configured to execute the computer-executable instructions and cause the processing system to:
receive machine learning model input data from a requestor;
process the machine learning model input data with a machine learning model using processing hardware at a first power level to generate first output data, wherein the processing hardware comprises a processor operable at one of a plurality of power levels including the first power level and a second power level different from the first power level, each power level corresponding to a different performance level for the machine learning model;
select the second power level for the processing hardware based on comparing the first output data to a threshold value;
process the machine learning model input data with the machine learning model using the processing hardware at the second power level to generate second output data; and
send second output data to the requestor, wherein:
the machine learning model comprises a neural network model, and
in order to compare the first output data to a threshold value, the one or more processors are further configured to cause the processing system to compare a difference of Softmax outputs for a first maximum candidate and a second maximum candidate to the threshold value.
19. A processing system for performing multimode inferencing, comprising:
a memory comprising computer-executable instructions;
one or more processors configured to execute the computer-executable instructions and cause the processing system to:
receive machine learning model input data from a requestor;
process the machine learning model input data with a machine learning model using processing hardware at a first power level to generate first output data, wherein the processing hardware comprises a processor operable at one of a plurality of power levels including the first power level and a second power level different from the first power level, each power level corresponding to a different performance level for the machine learning model;
select the second power level for the processing hardware based on comparing the first output data to a threshold value;
process the machine learning model input data with the machine learning model using the processing hardware at the second power level to generate second output data; and
send second output data to the requestor, wherein the one or more processors are further configured to cause the processing system to:
incrementing a counter value after selecting the second power level for the processing hardware; and
increasing the threshold value based on comparing the counter value to a counter threshold value.
20. A method for performing multimode inferencing, comprising:
receiving machine learning model input data from a requestor;
processing the machine learning model input data with a machine learning model using processing hardware at a first power level to generate first output data, wherein the machine learning model comprises a neural network model;
selecting a second power level for the processing hardware based on comparing the first output data to a threshold value, wherein comparing the first output data to a threshold value comprises at least one of:
comparing a maximum value of an activation function σ for the neural network model to the threshold value η according to: max σ(z) i max >η, where i max is an index for a final layer of the neural network model; and
comparing a difference of Softmax outputs for a first maximum candidate and a second maximum candidate to the threshold value;
processing the machine learning model input data with the machine learning model using the processing hardware at the second power level to generate second output data; and
sending second output data to the requestor.