Rapid predictive analysis of very large data sets using the distributed computational graph
A distributed computing cluster includes first, second, and third pluralities of computer systems. A first computer of the first plurality applies a first transformation pipeline to a stream of data to generate a output data, and transmits the output data to a computer of the second plurality, which is distinct from the first plurality. A second computer of the second plurality applies a second transformation pipeline. The second transformation pipeline includes a first storage transformation. A third computer of the third plurality stores a representation of a distributed computational graph (DCG), which includes a representation of a portion of the second transformation pipeline. The third computer processes the representation of the DCG, and determines whether the second transformation pipeline includes a storage transformation. The third computer monitors the second transformation pipeline, and in response, causes a fourth computer of the third plurality to apply a second storage transformation.
1. A distributed computing cluster comprising:
a first computer system of a first plurality of computer systems, wherein the first computer system is configured to execute software instructions that:
apply a first transformation pipeline to a first stream of data to generate a first stream of output data, and
transmit the first stream of output data to at least one computer system of a second plurality of computer systems, wherein the second plurality of computer systems is distinct from the first plurality of computer systems;
a second computer system of the second plurality of computer systems, wherein the second computer system is configured to execute software instructions that:
apply a second transformation pipeline to a second stream of data to generate a second stream of output data, wherein the second transformation pipeline comprises a first storage transformation;
a third computer system of a third plurality of computer systems, wherein the third computer system comprises a memory, and wherein the third computer system is configured to execute software instructions that:
store in the memory a representation of a distributed computational graph,
wherein the representation of the distributed computational graph comprises:
a representation of at least a portion of the first transformation pipeline,
a representation of at least a portion of the second transformation pipeline, and
a representation of at least a portion of a flow of data between the first transformation pipeline and the second transformation pipeline;
process the representation of the distributed computational graph,
based at least in part on said processing, determine whether the second transformation pipeline includes a storage transformation,
monitor said applying of the second transformation pipeline, and
in response to said monitoring, cause a fourth computer system of the third plurality of computer systems to execute software instructions that apply a second storage transformation.
2. The distributed computing cluster of claim 1 ,
wherein the second transformation pipeline further comprises a first transformation,
wherein the first transformation processes data stored by the first storage transformation; and,
wherein the third computer system is further configured to execute software instructions that:
in response to said monitoring, cause the fourth computer system to execute software instructions that apply a second transformation,
wherein the second transformation processes data stored by the second storage transformation.
3. The distributed computing cluster of claim 2 , wherein said causing the fourth computer system to execute software instructions that apply the second transformation comprises:
processing the representation of the distributed computational graph to determine a configuration of the first transformation, and
transmitting to the fourth computer system information describing a portion of the configuration.
4. The distributed computing cluster of claim 1 ,
wherein the second transformation pipeline further comprises a first transformation,
wherein the first storage transformation stores data processed by the first transformation; and,
wherein the third computer system is further configured to execute software instructions that:
in response to said monitoring, cause the fourth computer system to execute software instructions that apply a second transformation,
wherein the second storage transformation stores data processed by the second transformation.
5. The distributed computing cluster of claim 4 , wherein said causing the fourth computer system to execute software instructions that apply the second transformation comprises:
processing the representation of the distributed computational graph to determine a configuration of the first transformation, and
transmitting to the fourth computer system information describing a portion of the configuration.
6. The distributed computing cluster of claim 1 , wherein the second stream of data comprises at least a portion of the first stream of output data.
7. The distributed computing cluster of claim 1 , wherein said monitoring said applying the second transformation pipeline commences at least in part before said determining whether the second transformation pipeline includes at least a storage transformation.
8. The distributed computing cluster of claim 1 , wherein said monitoring said applying the second transformation pipeline commences at least in part before said processing the representation of the distributed computational graph.
9. The distributed computing cluster of claim 1 , wherein the at least one computer system of the second plurality of computer systems is the same computer system as the second computer system.
10. A distributed computing network comprising:
a first plurality of computer systems,
wherein each respective computer system of the first plurality of computer systems comprises a respective memory that stores a respective first data,
wherein the respective first data represents a respective portion of a first distributed computational graph, and
wherein the first distributed computational graph describes a first transformation pipeline; and,
wherein each respective computer system of the first plurality of computer systems is configured to:
receive a respective first stream of data from a feed,
process the respective first stream of data substantially in real time by executing respective software instructions that apply the first transformation pipeline to the respective first stream of data to generate a respective second stream of data, and
transmit the respective second stream of data to another computer system of the first plurality of computer systems, wherein the another computer system of the first plurality of computer systems is identified based on the first distributed computational graph;
a second plurality of computer systems distinct from the first plurality of computer systems, wherein each respective computer system of the second plurality of computer systems comprises a respective memory that stores a respective second data, and
a monitoring computer system that is part of either the first plurality of computer systems or the second plurality of computer systems, and wherein the monitoring computer system is configured to execute software instructions that:
monitor the execution of the first transformation pipeline,
in response to the monitoring, retrieve a representation of at least a portion of a second distributed computational graph, wherein the representation describes a flow of data from the first transformation pipeline to a second transformation pipeline,
process the representation, and
in response to the processing, cause at least one respective computer system of the second plurality of computer systems to process substantially in real time at least a respective portion of a third stream of data by applying the second transformation pipeline to the respective portion of the third stream of data,
wherein the third stream of data comprises at least a portion of at least one of the respective second streams of data.
11. The distributed computing network of claim 10 , wherein at least one of the first transformation pipeline and the second transformation pipeline is non-linear.
12. The distributed computing network of claim 11 , wherein the identified at least one respective computer system of the second plurality of computer systems is not configured to apply the second transformation pipeline to any stream of input data.
13. The distributed computing network of claim 11 , wherein the monitoring computer system is further configured to execute software instructions that:
identify one or more of the respective memories that store the respective first data representing at least a respective portion of the distributed computational graph,
wherein the respective first data includes information about the first transformation pipeline,
and store in the identified one or more memories information identifying at least one respective computer system of the second plurality of computer systems.
14. The distributed computing network of claim 10 , wherein the monitoring computer system is further configured to execute software instructions that:
identify one or more of the respective memories that store the respective first data representing at least a respective portion of the first distributed computational graph,
wherein the respective first data includes information about the second transformation pipeline,
and store in the identified one or more memories information identifying the another computer system of the first plurality of computer systems.
15. The distributed computing network of claim 10 , wherein the second plurality of computer systems are not executing any portion of any transformation pipeline in the flow of any of the distributed computational graphs.
16. The distributed computing network of claim 11 , wherein the second plurality of computer systems are not executing any portion of any transformation pipeline in the flow of any of the distributed computational graphs.