Rapid predictive analysis of very large data sets using the distributed computational graph
A system for predictive analysis of very large data sets using a distributed computational graph has been developed. Data receipt software receives streaming data from one or more sources. In a batch data pathway, data formalization software formats input data for storage. A batch event analysis server inspects stored data for trends, situations, or knowledge. Aggregated data is passed to message handler software. System sanity software receives status information from message handler and optimizes system performance. In the streaming pathway, transformation pipeline software manipulates the data stream, provides results back to the system, receives directives from the system sanity and retrain software.
1. A system comprising:
a distributed computing cluster comprising a first plurality of computer systems and a second plurality of computer systems,
wherein the first plurality of computer systems does not comprise any computer system of the second plurality of computer systems,
wherein the second plurality of computer systems does not comprise any computer system of the first plurality of computer systems,
wherein each respective computer system of the first plurality of computer systems comprises a memory that stores a respective first data,
wherein the respective first data represents a respective portion of a distributed computational graph,
and wherein the distributed computational graph describes a flow of output data of a first transformation pipeline to an input of a second transformation pipeline,
wherein a first computer system of the first plurality of computer systems is configured to:
receive a first stream of input data from an input feed,
process the first stream of input data substantially in real time by executing first software instructions that apply the first transformation pipeline to the first stream of input data to generate first pipeline output messages,
process at least a portion of the respective first data to determine information about the second transformation pipeline,
and transmit the first pipeline output messages to a second computer system of the first plurality of computer systems in accordance with the determined information,
wherein the second computer system is configured to:
receive the first pipeline output messages,
process the first pipeline output messages substantially in real time by executing second software instructions that apply the second transformation pipeline to the first pipeline output messages to generate second pipeline output messages,
process at least a portion of the respective first data to determine information about the second transformation pipeline,
and transmit the second pipeline output messages in accordance with the determined information,
wherein a third computer system of the second plurality of computer systems is configured to execute third software instructions that:
monitor a progress of the execution of at least one of the first software instructions and the second software instructions,
wherein the monitoring includes identifying a third transformation pipeline that has become needed,
modify the first data of a fourth computer system of the first plurality of computer systems such that the respective portion of the distributed computational graph described by the first data of the fourth computer system includes a flow of output data to an input of the third transformation pipeline,
and configure a fifth computer system of the second plurality of computer systems to process third pipeline output messages by executing fourth software instructions that apply the third transformation pipeline to the third pipeline output messages,
wherein the first and second computer systems are distinct.
2. The system of claim 1 , wherein monitoring the progress of the execution includes:
receiving at least one of the first pipeline output messages and the second pipeline output messages,
and employing machine learning algorithms in response to the receiving.
3. A system comprising:
a distributed computing cluster comprising a first plurality of computer systems and a second plurality of computer systems,
wherein each respective computer system of the first plurality of computer systems comprises a respective memory that stores respective first data that:
describes a configuration of a respective first transformation pipeline,
wherein a first computer system of the first plurality computer systems is configured to:
receive a first stream of data,
process the first stream of data substantially in real time by executing first software instructions that apply the first transformation pipeline of the memory of the first computer system to the first stream of data to generate a second stream of data,
and store, substantially in real time, at least a portion of data in the second stream of data in a database,
wherein a second computer system of the first plurality of computer systems is configured to execute second software instructions that:
monitor the execution of the first software instructions,
wherein the monitoring includes identifying a second transformation pipeline that has become needed,
configure a third computer system of the second plurality of computer systems to execute third software instructions that process the second stream of data by applying the second transformation pipeline to the second stream of data, to generate a third stream of data,
and store, substantially in real time, at least a portion of data in the third stream of data in the database.
4. The system of claim 3 , wherein monitoring the execution of the first software instructions includes:
monitoring at least a portion of the second stream of data,
and employing machine learning algorithms.
5. A system comprising:
a distributed computing cluster comprising a first plurality of computer systems and a second plurality of computer systems,
wherein each respective computer system of the first plurality of computer systems comprises a respective memory that stores respective first data that:
describes a configuration of a respective first transformation pipeline,
wherein a first computer system of the first plurality computer systems is configured to:
receive a first stream of data,
process the first stream of data substantially in real time by executing first software instructions that apply the first transformation pipeline of the memory of the first computer system to the first stream of data to generate a second stream of data,
and store, substantially in real time, at least a portion of data in the second stream of data in a database,
wherein a second computer system of the first plurality of computer systems is configured to execute second software instructions that:
monitor the execution of the first software instructions,
wherein the monitoring includes identifying a second transformation pipeline that has become needed,
configure a third computer system of the second plurality of computer systems to execute third software instructions that process a third stream of data by applying the second transformation pipeline to the third stream of data, to generate a fourth stream of data,
and store, substantially in real time, at least a portion of data in the fourth stream of data in the database.