Methods and apparatus for load balance optimization based on machine learning
An apparatus includes a processor, an operating system executed by the processor, and a memory storing code executed by the processor to receive performance data from the operating system and from other compute devices. The apparatus includes a machine learning model trained with the performance data. The apparatus uses the machine learning model to predict workload values of the apparatus and other compute devices. The workload values are predicted for a future time window. The apparatus commands an execution of a data transformation task of a first dataset, based on the predicted workload values and criteria to reduce time consumed in the execution of the data transformation task. Thereafter, the apparatus receives a notification signal indicative of a completion of the data transformation task, and an indicator associated with a second dataset different from the first dataset, produced from the execution of the data transformation task.
1. An apparatus, comprising:
a processor;
an operating system executed by the processor; and
a memory storing code which, when executed by the processor, causes the processors to:
receive, at the processor, performance data from the operating system and a plurality of compute devices;
predict, based on a regression machine learning model trained with the received performance data, a first workload value and a second workload value, the first workload value indicating a workload measure of the apparatus predicted for a future time window, the second workload value indicating a workload measure of a compute device from the plurality of compute devices predicted for the future time window;
command an execution of a data transformation task of a first dataset to one of the apparatus or the compute device, based on (1) criteria to reduce time consumed in the data transformation task, (2) the first workload value, and (3) the second workload value; and
receive a notification signal indicative of a completion of the transformation task, and an indicator associated with a second dataset different from the first dataset and produced from the execution of the data transformation task.
2. The apparatus of claim 1 , wherein the code to command the execution of the data transformation task includes code to:
execute the data transformation task at the apparatus when the first workload value is less than the second workload value.
3. The apparatus of claim 1 , wherein the code to command the execution of the data transformation task includes code to:
send a command to execute the data transformation task to the compute device when the first workload value is greater or equal to the second workload value.
4. The apparatus of claim 1 , wherein the code to command the execution of the data transformation task includes code to:
send a parameter with the data transformation task indicative of a type of data transformation operation to be performed on the first dataset, the parameter indicative of at least one of (1) a data format conversion operation; (2) a data sorting operation; (3) a data summing operation; (4) a data averaging operation; and (5) a data sampling operation.
5. The apparatus of claim 1 , wherein the code to command the execution of the data transformation task includes code to:
send a parameter indicating an endpoint address to access the first dataset.
6. The apparatus of claim 1 , wherein the indicator associated with the second dataset is an endpoint address to access the second dataset.
7. The apparatus of claim 1 , wherein the indicator associated with the second dataset is a file including the second dataset.
8. The apparatus of claim 1 , wherein the regression machine learning model is a time series non-linear regression machine learning model trained to predict workload values associated with data transformation tasks.
9. The apparatus of claim 1 , wherein the performance data includes performance data from the operating system sampled during execution of a plurality of completed data transformation tasks.
10. An apparatus, comprising:
a processor; and
a memory storing code which, when executed by the processor, causes the processor to:
instantiate an application configured to receive, from at least one compute device, a plurality of requests to execute data transformation tasks;
predict a readiness value of the apparatus for a future time window, based on a regression machine learning model trained, at least in part, with a first set of requests from the plurality of requests, the readiness value indicating a measure of capacity of the apparatus to execute data transformation tasks;
accept a request from a second set of requests from the plurality of requests, based on (1) the readiness value and (2) criteria to improve the readiness value according to at least one task priority order of the data transformation tasks, the request associated with a first dataset;
produce, in response to the accepted request, via the application, a second dataset different from the first dataset; and
send the second dataset to a compute device from the at least one compute device.
11. The apparatus of claim 10 , wherein the code further causes the processor to:
send the readiness value to the at least one compute device; and
receive the second set of requests from the plurality of requests in response to the readiness value.
12. The apparatus of claim 10 , wherein the code further causes the processor to:
train the regression machine learning model with a training set including workload values sampled from the processor during execution of requested data transformation tasks included in the first set of requests.
13. The apparatus of claim 10 , wherein the code further causes the processor to:
train the regression machine learning model with a time series including a time window identifier and an average workload value of the apparatus, sampled from the apparatus during the execution of the first set of requests.
14. The apparatus of claim 10 , wherein the code to produce, in response to the accepted request, via the application, a second dataset includes code to:
produce the second dataset within a time period bounded by the future time window.
15. The apparatus of claim 10 , wherein the task priority order is based on priorities given to compute devices from the at least one compute device.
16. The apparatus of claim 10 , wherein the task priority order is based on priorities given to types of data transformation tasks included in the plurality of requests to execute data transformation tasks.
17. A method, comprising:
receiving, at a processor of a first compute device, performance data from a plurality of compute devices;
triggering a first set of rules, each rule from the first set of rules includes a distinct antecedent pattern satisfied by the performance data received from the plurality of compute devices;
firing a second set of rules, based on criteria to improve a performance metric value of a second compute device from the plurality of compute devices, the second set of rules selected from the first set of rules to produce at least one rule consequent;
determining, based on the at least one rule consequent, a third compute device from the plurality of compute devices for the execution of a data transformation task;
sending a command to the third compute device to execute the data transformation task, the data transformation task associated with a first dataset; and
receiving from the third compute device a second dataset different from the first dataset in response to the command.
18. The method of claim 17 , wherein the plurality of compute devices includes a mainframe and the second compute device is the mainframe.
19. The method of claim 17 , wherein the distinct antecedent pattern includes one or more of: (1) an amount of memory unused by the third compute device, (2) processor characteristics of the third compute device, and (3) historical workload patterns of the third compute device.
20. The method of claim 17 , wherein the distinct antecedent pattern includes data indicative of one or more of: (1) whether the data transformation task is critical, (2) whether a workload value of the third compute device corresponds to an ordinal value, (3) whether the data transformation task requires processing a file of a size that is below a threshold file size, (4) whether an application requesting the data transformation task is a prioritized application, and (6) whether the data transformation task can be postponed.