SYSTEM AND METHOD FOR OPTIMIZATION AND LOAD BALANCING OF COMPUTER CLUSTERS
A system and methods for optimization and load balancing for computer clusters, comprising a distributed computational graph, a server architecture using multi-dimensional time-series databases for continuous load simulation and forecasting, a server architecture using traditional databases for discrete load simulation and forecasting, and using a combination of real-time data and records of previous activity for continuous and precise load forecasting for computer clusters, datacenters, or servers.
1 . A system for optimization and load balancing for computer clusters, comprising:
a distributed computational graph module comprising a first plurality of programming instructions stored in the memory and operating on the processor, wherein the first plurality of programming instructions, when operating on the processor, causes the processor to:
receive a distributed computational graph defining a data processing workflow wherein:
the vertices of the distributed computational graph represent data transformation stages and the edges of the distributed computational graph represent messaging between the data transformation stages; and
the data processing workflow comprises one or more data pipelines for analysis of load balancing of a plurality of devices, each data pipeline comprising a series of nodes and edges of the directed computational graph;
maintain a plurality of connections with each of the plurality of devices over a network, wherein each connection provides the ability to send data to, and receive data from, the respective device over the network; and
analyze a dataset using the data pipeline as requested by a load forecasting application to produce a load balancing result;
the load forecasting application comprising a second plurality of programming instructions stored in the memory and operating on the processor, wherein the second plurality of programming instructions, when operating on the processor, causes the processor to:
query a multidimensional time-series database for portions of the recorded data for one or more of the plurality of devices;
utilize the received response data to produce a load simulation, wherein the load simulation comprises a the distributed computational graph;
provide the distributed computational graph to the directed computational graph module for execution;
receive the load balancing result from the directed computational graph; and
create a redistribution of a processing load among the plurality of devices.
2 . The system of claim 1 , wherein the load forecasting application queries data from a database other than a multidimensional time-series database.
3 . The system of claim 1 , wherein the load forecasting application operates on continuous data from a multidimensional time-series database operating on the same computing device as the load forecasting application.
4 . The system of claim 1 , wherein the load forecasting application operates on continuous data from a multidimensional time-series database operating on a device connected by a network.