System and method for optimizing and load balancing of applications using distributed computer clusters
A system and method have been devised for optimization and load balancing for computer clusters, comprising a distributed computational graph, a server architecture using multi-dimensional time-series databases for continuous load simulation and forecasting, a server architecture using traditional databases for discrete load simulation and forecasting, and using a combination of real-time data and records of previous activity for continuous and precise load forecasting for computer clusters, datacenters, or servers.
1. A system for optimization and load balancing for applications using distributed computer clusters, comprising:
a computing device comprising a memory, a processor, and a non-volatile data storage device;
a distributed computational graph module comprising a first plurality of programming instructions stored in the memory and operating on the processor, wherein the first plurality of programming instructions, when operating on the processor, causes the computing device to:
receive a distributed computational graph defining a data processing workflow wherein:
each vertex of the distributed computational graph represents an application-specific task, the weight of the vertex corresponding to an expected computational cost, the edges of the distributed computational graph represent a task dependency, and the weight of the edges corresponding to an expected communication cost; and
the data processing workflow comprises one or more data pipelines for analysis of load balancing of a plurality of tasks on a plurality of devices, each data pipeline comprising a series of nodes and edges of the directed computational graph;
maintain a plurality of connections with each of the plurality of devices over a network,
wherein each connection provides the ability to send data to, and receive data from, the respective device over the network; and
analyze a dataset using the one or more data pipelines as requested by a load forecasting application to produce a mapping of grouped application-specific tasks to a plurality cloud computing resources;
a multidimensional time-series database module comprising a second plurality of programming instructions stored in the memory and operating on the processor, wherein the second plurality of programming instructions, when operating on the processor, causes the computing device to:
record received data from the plurality of devices according to a user configuration, wherein the recording occurs continuously over time and wherein the recorded data comprises both the received data from each of the plurality of devices and a time stamp describing when the data was originally received from each device; and
provide response data comprising portions of the recorded data as requested by the load forecasting application; and
the load forecasting application comprising a third plurality of programming instructions stored in the memory and operating on the processor, wherein the third plurality of programming instructions, when operating on the processor, causes the computing device to:
query the multidimensional time-series database for portions of the recorded data for one or more of the plurality of devices;
utilize the response data received from the multidimensional time-series database to produce a load simulation, wherein the load simulation comprises the distributed computational graph;
provide the distributed computational graph to the directed computational graph module for determining a mapping of tasks to the plurality of cloud computing resources according to the load simulation;
retrieve the mapping from the directed computational graph; and
redistribute each application-specific task to the cloud computing resource according to the mapping.
2. The system of claim 1 , wherein the load forecasting application queries data from a database other than the multidimensional time-series database.
3. The system of claim 1 , wherein the load forecasting application operates on continuous data from a multidimensional time-series database operating on the same computing device as the load forecasting application.
4. The system of claim 1 , wherein the load forecasting application operates on continuous data from a multidimensional time-series database operating on a device connected by a network.
5. The system of claim 1 , wherein each application-specific task in running in a virtualized environment.
6. The system of claim 5 , wherein the load forecasting application redistributes each virtualized environment to the cloud computing resource according to the mapping.
7. A method for optimization and load balancing for applications using distributed computer clusters, comprising the steps of:
receiving a distributed computational graph defining a data processing workflow wherein:
each vertex of the distributed computational graph represents an application-specific task, the weight of the vertex corresponding to an expected computational cost, the edges of the distributed computational graph represent a task dependency, and the weight of the edges corresponding to an expected communication cost; and
the data processing workflow comprises one or more data pipelines for analysis of load balancing of a plurality of tasks on devices, each data pipeline comprising a series of nodes and edges of the directed computational graph;
maintaining a plurality of connections with each of the plurality of devices over a network, wherein each connection provides the ability to send data to, and receive data from, the respective device over the network;
analyzing a dataset using the one or more data pipelines as requested by a load forecasting application to produce a mapping of grouped application-specific tasks to a plurality cloud computing resources;
recording received data from the plurality of devices according to a user configuration, wherein the recording occurs continuously over time and wherein the recorded data comprises both the received data from each of the plurality of devices and a time stamp describing when the data was originally received from each device;
producing a load simulation, wherein the load simulation comprises the distributed computational graph;
determining a mapping of tasks to the plurality of cloud computing resources according to the load simulation; and
redistributing each application-specific task to the cloud computing resource according to the mapping.
8. The method of claim 7 , wherein each application-specific task in running in a virtualized environment.
9. The method of claim 7 , wherein the load forecasting application redistributes each virtualized environment to the cloud computing resource according to the mapping.