Rapid predictive analysis of very large data sets using the distributed computational graph
A system for predictive analysis of very large data sets using a distributed computational graph has been developed. Data receipt software receives streaming data from one or more sources. In a batch data pathway, data formalization software formats input data for storage. A batch event analysis server inspects stored data for trends, situations, or knowledge. Aggregated data is passed to message handler software. System sanity software receives status information from message handler and optimizes system performance. In the streaming pathway, transformation pipeline software manipulates the data stream, provides results back to the system, receives directives from the system sanity and retrain software.
1. A distributed computing cluster comprising:
a first plurality of computer systems,
wherein each respective computer system of the first plurality of computer systems comprises a memory that stores a respective first data,
wherein the respective first data represents a respective portion of a distributed computational graph,
and wherein the distributed computational graph describes a flow of output data of a first transformation pipeline to an input of a second transformation pipeline,
wherein a first computer system of the first plurality of computer systems is configured to:
receive a first stream of input data from a first input feed,
process the first stream of input data substantially in real time by executing software instructions that apply the first transformation pipeline to the first stream of input data to generate first pipeline output messages,
process the respective first data stored in the memory of the first computer system to determine information about the second transformation pipeline,
and transmit the first pipeline output messages to a second computer system of the first plurality of computer systems in accordance with the determined information,
wherein the second computer system is configured to:
receive the first pipeline output messages,
and process the first pipeline output messages substantially in real time by executing software instructions that apply the second transformation pipeline to the first pipeline output messages to generate second pipeline output messages,
wherein the first and second computer systems are distinct; and
a second plurality of computer systems;
wherein a third computer system of the first plurality of computer systems is configured to execute software instructions that cause a fourth computer system of the second plurality of computer systems to execute software instructions that apply at least one of the first transformation pipeline and the second transformation pipeline.
2. The distributed computing cluster of claim 1 , wherein at least one of the first transformation pipeline and the second transformation pipeline is non-linear.
3. The distributed computing cluster of claim 2 ,
wherein the third computer system of the first plurality of computer systems is configured to:
identify a fifth computer system of the first plurality of computer systems,
wherein the fifth computer system is not configured to apply the first transformation pipeline to any stream of input data,
and cause the fifth computer system to process a second stream of input data received from a second input feed substantially in real time by executing software instructions that apply the first transformation pipeline to the second stream of input data.
4. The distributed computing cluster of claim 3 , wherein the third computer system is further configured to execute software instructions that:
identify one or more of the respective memories that store respective first data representing at least a respective portion of the distributed computational graph,
wherein the respective first data includes information about the first transformation pipeline,
and store in the identified one or more memories information identifying the fifth computer system.
5. The distributed computing cluster of claim 2 ,
wherein the third computer system of the first plurality of computer systems is configured to:
identify a fifth computer system of the first plurality of computer systems,
wherein the fifth computer system is not configured to apply the second transformation pipeline to any pipeline output message,
and cause the fifth computer system to process third pipeline output messages substantially in real time by executing software instructions that apply the second transformation pipeline to the third pipeline output messages.
6. The distributed computing cluster of claim 5 , wherein the third computer system is further configured to execute software instructions that:
identify one or more of the respective memories that store respective first data representing at least a respective portion of the distributed computational graph,
wherein the respective first data includes information about the second transformation pipeline,
and store in the identified one or more memories information identifying the fifth computer system.
7. The distributed computing cluster of claim 1 ,
wherein the first computer system is further configured to transmit the first pipeline output messages to the fourth computer system of the second plurality of computer systems,
wherein the first computer system is distinct from the fourth computer system,
and wherein the second computer system is distinct from the fourth computer system.
8. The distributed computing cluster of claim 1 ,
wherein the first computer system is further configured to transmit the first pipeline output messages to another computer system of the second plurality of computer systems.
9. A distributed computing cluster comprising:
a first plurality of computer systems,
wherein each respective computer system of the first plurality of computer systems comprises a respective memory that stores a respective first data that:
describes at least a respective portion of a configuration of a first transformation pipeline,
and identifies at least a second transformation pipeline,
wherein a first computer system of the first plurality of computer systems is configured to:
receive a first stream of input data from a first input feed,
process the first stream of input data substantially in real time by executing software instructions that apply the first transformation pipeline to the first stream of input data to generate first pipeline output messages,
process the respective first data to determine an identification of the second transformation pipeline,
and transmit the first pipeline output messages to a second computer system of the first plurality of computer systems in accordance with the determined identification,
wherein the second computer system is configured to:
receive the first pipeline output messages,
and process the first pipeline output messages substantially in real time by executing software instructions that apply the second transformation pipeline to the first pipeline output messages,
wherein the first and second computer systems are distinct; and
a second plurality of computer systems;
wherein a third computer system of the first plurality of computer systems is configured to execute software instructions that cause the second plurality of computer systems to execute at least one of the first transformation pipeline and the second transformation pipeline.
10. The distributed computing cluster of claim 9 , wherein one or more of the first transformation pipeline and the second transformation pipeline is non-linear.
11. The distributed computing cluster of claim 10 ,
wherein the third computer system of the first plurality of computer systems is configured to:
identify a fourth computer system of the first plurality of computer systems,
wherein the fourth computer system is not configured to apply the first transformation pipeline to any stream of input data,
and cause the fourth computer system to process a second stream of input data received from a second input feed substantially in real time by executing software instructions that apply the first transformation pipeline to the second stream of input data.
12. The distributed computing cluster of claim 11 , wherein the third computer system is further configured to execute software instructions that:
identify one or more of the respective memories that stores respective data that describes a respective portion of a configuration of the first transformation pipeline,
and store in the identified one or more memories information identifying the fourth computer system.
13. The distributed computing cluster of claim 9 ,
wherein the first computer system is further configured to transmit the first pipeline output messages to a fourth computer system of the second plurality of computer systems,
wherein the first computer system is distinct from the fourth computer system,
and wherein the second computer system is distinct from the fourth computer system.