System and method for optimization and load balancing of computer clusters
A system and methods for optimization and load balancing for computer clusters, comprising a distributed computational graph, a server architecture using multi-dimensional time-series databases for continuous load simulation and forecasting, a server architecture using traditional databases for discrete load simulation and forecasting, and using a combination of real-time data and records of previous activity for continuous and precise load forecasting for computer clusters, datacenters, or servers.
1. A system for optimization and load balancing for computer clusters, comprising:
a computing device comprising a memory, a processor, and a non-volatile data storage device;
a distributed computational graph module comprising a first plurality of programming instructions stored in the memory and operating on the processor, wherein the first plurality of programming instructions, when operating on the processor, causes the computing device to:
receive a distributed computational graph defining a data processing workflow wherein:
the vertices of the distributed computational graph represent data transformation stages and the edges of the distributed computational graph represent messaging between the data transformation stages; and
the data processing workflow comprises one or more data pipelines for analysis of load balancing of a plurality of devices, each data pipeline comprising a series of nodes and edges of the directed computational graph;
maintain a plurality of connections with each of the plurality of devices over a network, wherein each connection provides the ability to send data to, and receive data from, the respective device over the network;
analyze a dataset using the data pipeline as requested by a load forecasting application to produce a load balancing result;
a multidimensional time-series database module comprising a second plurality of programming instructions stored in the memory and operating on the processor, wherein the second plurality of programming instructions, when operating on the processor, causes the computing device to:
record received data from the plurality of devices according to a user configuration, wherein the recording occurs continuously over time and wherein the recorded data comprises both the received data from each of the plurality of devices and a time stamp describing when the data was originally received from each device;
provide response data comprising portions of the recorded data as requested by the load forecasting application; and
the load forecasting application comprising a third plurality of programming instructions stored in the memory and operating on the processor, wherein the third plurality of programming instructions, when operating on the processor, causes the computing device to:
query the multidimensional time-series database for portions of the recorded data for one or more of the plurality of devices;
utilize the response data received from the multidimensional time-series database to produce a load simulation, wherein the load simulation comprises a the distributed computational graph;
provide the distributed computational graph to the directed computational graph module for execution;
receive the load balancing result from the directed computational graph; and
create a redistribution of a processing load among the plurality of devices.
2. The system of claim 1 , wherein the load forecasting application queries data from a database other than the multidimensional time-series database.
3. The system of claim 1 , wherein the load forecasting application operates on continuous data from a multidimensional time-series database operating on the same computing device as the load forecasting application.
4. The system of claim 1 , wherein the load forecasting application operates on continuous data from a multidimensional time-series database operating on a device connected by a network.