Distributing workloads to hardware accelerators during transient workload spikes
Systems and methods are provided for strategically harvesting untapped compute capacity of hardware accelerators to manage transient workload spikes at computing systems, are provided. Examples provide a low-cost and scalable computing system which orchestrates seamless offloading of workloads to hardware accelerators during transient workload spikes. By utilizing hardware accelerators as short-term emergency buffers, examples improve upon existing approaches which deploy more expensive, and often significantly under-utilized servers for these emergency purposes. Accordingly, examples may reduce the occurrence of SLA violations while minimizing capital expenditure in computing power.
1 . A method comprising:
predicting a transient workload spike based on monitored historical data regarding past workloads received by a computing system, wherein the computing system includes a server and a hardware accelerator, and wherein, prior to the transient workload spike, a service time of the server in responding to incoming workloads is less than a service time of the hardware accelerator in responding to the incoming workloads, wherein a window-based prediction model is used to predict the transient workload spike and a window size of the window-based prediction model is increased when workload variation in the monitored historical data is less than a first percentage and is decreased when the workload variation is more than a second percentage, wherein the first percentage is less than the second percentage;
monitoring values of the service time of the server in servicing the incoming workloads distributed to the server;
predicting that a value of the service time of the server will exceed a threshold value for the service time at a time prior to or during the predicted transient workload spike;
responsive to predicting that the value of the service time will exceed the threshold value for the service time, determining that the service time of the server, in responding to the incoming workloads distributed to the server, will exceed the service time of the hardware accelerator during the predicted transient workload spike; and
responsive to the determination, offloading at least one incoming workloads of the incoming workload distributed to the server to the hardware accelerator and executing the at least one incoming workload on the hardware accelerator.
2 . The method of claim 1 , wherein the hardware accelerator comprises a System on a Chip (SOC) based Smart Network Interface Card (SmartNIC).
3 . The method of claim 1 , wherein the monitored historical data is specific to an application and distributing the at least one incoming workload is for the application.
4 . The method of claim 3 , wherein the threshold value for the service time corresponds to a specification in a service level agreement for the application.
5 . The method of claim 1 , wherein:
the past workloads received by the computing system are past serverless queries received by the computing system;
the at least one incoming workload is at least one incoming serverless query; and
the at least one incoming serverless query is executed within a workload container at the hardware accelerator.
6 . The method of claim 5 , wherein the workload container at the hardware accelerator is started before the predicted transient workload spike.
7 . The method of claim 6 , wherein the starting the workload container comprising starting a containerized runtime environment on the hardware accelerator prior to arrival of the predicted transient spike.
8 . The method of claim 1 , wherein the transient workload spike is predicted based on a support vector regression (SVR) prediction model.
9 . A computing system comprising:
a plurality of processing resources associated with the computing system; and
a non-transitory computer-readable medium, coupled to the plurality of processing resources, having stored therein instructions that when executed by the processing resources cause the computing system to:
predict a transient workload spike based on monitored historical data regarding past workloads received by the computing system, wherein the computing system includes a server and a hardware accelerator, and wherein, prior to the transient workload spike, a service time of the server in responding to incoming workloads is less than a service time of the hardware accelerator in responding to the incoming workloads, wherein a window-based prediction model is used to predict the transient workload spike and a window size of the window-based prediction model is increased when workload variation in the monitored historical data is less than a first percentage and is decreased when the workload variation is more than a second percentage, wherein the first percentage is less than the second percentage;
predict that a value of a service time of the computing system will exceed a threshold value of the service time at a time prior to or during the predicted transient workload spike unless at least one incoming workload is distributed to the hardware accelerator;
start a workload container at the hardware accelerator prior to the predicted transient workload spike; and
based on responsive to predicting that the value of the service time will exceed the threshold value for the service time, determining that the service time of the server will exceed the service time of the hardware accelerator during the predicted transient workload spike; and
responsive to the determination, offloading at least one incoming workload distributed to the server to a workload container at the hardware accelerator and executing the at least one incoming workload by the workload container at the hardware accelerator.
10 . The computing system of claim 9 , wherein the threshold value for the service time corresponds to a specification in a service level agreement.
11 . The computing system of claim 9 , wherein the hardware accelerator comprises a network accelerator.
12 . The computing system of claim 11 , wherein the network accelerator comprises a System on a Chip (SOC) based Smart Network Interface Card (SmartNIC).
13 . The computing system of claim 9 , wherein the transient workload spike is predicted based on a support vector regression (SVR) prediction model.
14 . The computing system of claim 9 , wherein the monitored historical data is specific to an application and distributing the at least one incoming workload is for the application.
15 . A non-transitory computer-readable medium storing instructions, which when executed by a plurality of processing resources of an edge-computing system, cause the edge-computing system to:
receive a query from an Application Programming Interface (API) gateway of the edge-computing system, wherein the edge-computing system includes an edge server and a hardware accelerator;
predict a transient workload spike using a window-based prediction model, a window size of the window-based prediction model is increased when workload variation in monitored historical data is less than a first percentage and is decreased when the workload variation is more than a second percentage, wherein the first percentage is less than the second percentage;
based on a determination that the transient workload spike exceeds a threshold value of a service time of the edge server, determine a distribution of queries over a time horizon which includes the predicted transient workload spike, and determine that, during the predicted transient workload spike, the service time of the edge server in responding to the distribution of queries will exceed a service time of the hardware accelerator in responding to the distribution of queries, wherein the service time of the edge server is less than the service time of the hardware accelerator prior to the transient workload spike; and
based on the determined distribution of queries and on the determination that the service time of the edge server will exceed the service time of the hardware accelerator during the predicted transient workload spike, offload the query from the edge server to a workload container at the hardware accelerator and execute the query by the workload container at the hardware accelerator.
16 . The non-transitory computer-readable medium of claim 15 , wherein the hardware accelerator comprises a System on a Chip (SOC) based Smart Network Interface Card (SmartNIC).
17 . The non-transitory computer-readable medium of claim 15 , wherein the workload container at the hardware accelerator is started before the predicted transient workload spike.
18 . The non-transitory computer-readable medium of claim 17 , wherein the monitored historical data is specific to an application.