Massively scalable computer platform using a distributed computational graph
A system for repeatable and flexible predictive analysis of very large data sets using a distributed computational graph using a domain specific language for pipeline and resource declaration has been developed. Data receipt software receives data from one or more sources. In a declared data pathway, data formalization software formats input data for storage, enrichment, or continued processing. An event analysis server inspects stored data for trends, situations, or knowledge. Aggregated data is passed to event message store and data stream handler software. Transformation pipeline software stores, triggers, executes and monitors declared transformation steps and pipelines to manipulate and act on bulk data and data streams, provides results and enables dynamic model and routine re-training and learning.
1 . A computing system for a massively scalable compute platform, the computing system comprising:
one or more hardware processors configured for:
receiving streams of input from one or more of a plurality of data sources;
filtering data from the streams of input based on data validity rules;
instantiating, based on a pre-defined resource declaration in a domain-specific language, a graph comprising a plurality of data transformation pipelines, wherein each data transformation pipeline comprises a plurality of nodes implemented across a plurality of distributed computing devices, each node representing one or more functions that act on input data and produce output data and further comprises a plurality of edges that represent messages carrying data between individual nodes;
processing the streams of input using the graph; and
during processing of the streams of input:
analyzing operational performance of one or more nodes of the graph;
reconfiguring one or more of the data transformation pipelines based on the resource declaration in the domain-specific language in response to the analyzed operational performance; and
sending one or more output streams of transformed data to requesting devices on a network.
2 . The computing system of claim 1 , wherein the one or more hardware processors are further configured for:
formatting the data within the streams of input based upon a set of predetermined parameters to prepare the data for storage in an input event data store; and
performing batch or stream analysis of stored data from the input event data store to determine at least one of: a set of trends in the stored data, a previous occurrence of a progression of a set of events found in the stored data, or an occurrence of a single specific event in the stored data.
3 . The computing system of claim 1 , wherein multiple antecedent node outputs are used as input to one node.
4 . The computing system of claim 1 , wherein output from one node acts as input to one or more downstream nodes.
5 . The computing system of claim 1 , wherein one or more of the data transformation pipelines comprises a cyclical subgraph configuration.
6 . The computing system of claim 1 , wherein the structure of one or more of the data transformation pipelines is a directed graph.
7 . The computing system of claim 1 , wherein at least one node within one or more of the data transformation pipelines operates as a data store and forms a queue for subsequent nodes to serialize effects of the one or more functions at branches of said one or more data transformation pipelines or at a head of a cyclical configuration of said one or more data transformation pipelines.
8 . The computing system of claim 1 , wherein the analysis of the operational performance of the one or more nodes of the graph is based on collected administrative execution and operational data associated with the graph; and
wherein the collected administrative execution and operational data associated with the graph comprises at least one of: execution data, failure data, trigger data, and timing data.
9 . The computing system of claim 8 , wherein the collected administrative execution and operational data associated with the graph further comprises at least one of: resource usage and input or output data.
10 . A computer-implemented method executed on a massively scalable compute platform, the computer-implemented method comprising:
receiving streams of input from one or more of a plurality of data sources;
filtering data from the streams of input based on data validity rules;
instantiating, based on a pre-defined resource declaration in a domain-specific language, a graph comprising a plurality of data transformation pipelines, wherein each data transformation pipeline comprises a plurality of nodes implemented across a plurality of distributed computing devices, each node representing one or more functions that act on input data and produce output data and further comprises a plurality of edges that represent messages carrying data between individual nodes;
processing the streams of input using the graph; and
during processing of the streams of input:
analyzing operational performance of one or more nodes of the graph;
reconfiguring one or more of the data transformation pipelines based on the resource declaration in the domain-specific language in response to the analyzed operational performance; and
sending one or more output streams of transformed data to requesting devices on a network.
11 . The computer-implemented method of claim 10 , further comprising the steps of:
formatting the data within the streams of input based upon a set of predetermined parameters to prepare the data for storage in an input event data store; and
performing batch or stream analysis of stored data from the input event data store to determine at least one of: a set of trends in the stored data, a previous occurrence of a progression of a set of events found in the stored data, or an occurrence of a single specific event in the stored data.
12 . The computer-implemented method of claim 10 , wherein multiple antecedent node outputs are used as input to one node.
13 . The computer-implemented method of claim 10 , wherein output from one node acts as input to one or more downstream nodes.
14 . The computer-implemented method of claim 10 , wherein one or more of the data transformation pipelines comprises a cyclical subgraph configuration.
15 . The computer-implemented method of claim 10 , wherein the structure of one or more of the data transformation pipelines is a directed graph.
16 . The computer-implemented method of claim 10 , wherein at least one node within one or more of the data transformation pipelines operates as a data store and forms a queue for subsequent nodes to serialize effects of the one or more functions at branches of said one or more data transformation pipelines or at a head of a cyclical configuration of said one or more data transformation pipelines.
17 . The computer-implemented method of claim 10 , wherein the analysis of the operational performance of the one or more nodes of the graph is based on collected administrative execution and operational data associated with the graph; and
wherein the collected administrative execution and operational data associated with the graph comprises at least one of: execution data, failure data, trigger data, and timing data.
18 . The computer-implemented method of claim 17 , wherein the collected administrative execution and operational data associated with the graph further comprises at least one of: resource usage and input or output data.
19 . A system for a massively scalable compute platform, comprising one or more computers with executable instructions that, when executed, cause the system to:
receive streams of input from one or more of a plurality of data sources;
filter data from the streams of input based on data validity rules;
instantiate, based on a pre-defined resource declaration in a domain-specific language, a graph comprising a plurality of data transformation pipelines, wherein each data transformation pipeline comprises a plurality of nodes implemented across a plurality of distributed computing devices, each node representing one or more functions that act on input data and produce output data and further comprises a plurality of edges that represent messages carrying data between individual nodes;
process the streams of input using the graph; and
during processing of the streams of input:
analyze operational performance of one or more nodes of the graph;
reconfigure one or more of the data transformation pipelines based on the resource declaration in the domain-specific language in response to the analyzed operational performance; and
send one or more output streams of transformed data to requesting devices on a network.
20 . The system of claim 19 , wherein the system is further caused to:
format the data within the streams of input based upon a set of predetermined parameters to prepare the data for storage in an input event data store; and
perform batch or stream analysis of stored data from the input event data store to determine at least one of: a set of trends in the stored data, a previous occurrence of a progression of a set of events found in the stored data, or an occurrence of a single specific event in the stored data.
21 . The system of claim 19 , wherein multiple antecedent node outputs are used as input to one node.
22 . The system of claim 19 , wherein output from one node acts as input to one or more downstream nodes.
23 . The system of claim 19 , wherein one or more of the data transformation pipelines comprises a cyclical subgraph configuration.
24 . The system of claim 19 , wherein the structure of one or more of the data transformation pipelines is a directed graph.
25 . The system of claim 19 , wherein at least one node within one or more of the data transformation pipelines operates as a data store and forms a queue for subsequent nodes to serialize effects of the one or more functions at branches of said one or more data transformation pipelines or at a head of a cyclical configuration of said one or more data transformation pipelines.
26 . The system of claim 19 , wherein the analysis of the operational performance of the one or more nodes of the graph is based on collected administrative execution and operational data associated with the graph; and
wherein the collected administrative execution and operational data associated with the graph comprises at least one of: execution data, failure data, trigger data, and timing data.
27 . The system of claim 26 , wherein the collected administrative execution and operational data associated with the graph further comprises at least one of:
resource usage and input or output data.
28 . Non-transitory, computer-readable storage media having computer executable instructions embodied thereon that, when executed by one or more processors of a computing system employing a massively scalable compute platform, cause the computing system to:
receive streams of input from one or more of a plurality of data sources;
filter data from the streams of input based on data validity rules;
instantiate, based on a pre-defined resource declaration in a domain-specific language, a graph comprising a plurality of data transformation pipelines, wherein each data transformation pipeline comprises a plurality of nodes implemented across a plurality of distributed computing devices, each node representing one or more functions that act on input data and produce output data and further comprises a plurality of edges that represent messages carrying data between individual nodes;
process the streams of input using the graph; and
during processing of the streams of input:
analyze operational performance of one or more nodes of the graph;
reconfigure one or more of the data transformation pipelines based on the resource declaration in the domain-specific language in response to the analyzed operational performance; and
send one or more output streams of transformed data to requesting devices on a network.
29 . The non-transitory, computer-readable storage media of claim 28 , wherein the computing system is further caused to:
format the data within the streams of input based upon a set of predetermined parameters to prepare the data for storage in an input event data store; and
perform batch or stream analysis of stored data from the input event data store to determine at least one of: a set of trends in the stored data, a previous occurrence of a progression of a set of events found in the stored data, or an occurrence of a single specific event in the stored data.
30 . The non-transitory, computer-readable storage media of claim 28 , wherein multiple antecedent node outputs are used as input to one node.
31 . The non-transitory, computer-readable storage media of claim 28 , wherein output from one node acts as input to one or more downstream nodes.
32 . The non-transitory, computer-readable storage media of claim 28 , wherein one or more of the data transformation pipelines comprises a cyclical subgraph configuration.
33 . The non-transitory, computer-readable storage media of claim 28 wherein the structure of one or more of the data transformation pipelines is a directed graph.
34 . The non-transitory, computer-readable storage media of claim 28 , wherein at least one node within one or more of the data transformation pipelines operates as a data store and forms a queue for subsequent nodes to serialize effects of the one or more functions at branches of said one or more data transformation pipelines or at a head of a cyclical configuration of said one or more data transformation pipelines.
35 . The non-transitory, computer-readable storage media of claim 28 , wherein the analysis of the operational performance of the one or more nodes of the graph is based on collected administrative execution and operational data associated with the graph; and
wherein the collected administrative execution and operational data associated with the graph comprises at least one of: execution data, failure data, trigger data, and timing data.
36 . The non-transitory, computer-readable storage media of claim 35 , wherein the collected administrative execution and operational data associated with the graph further comprises at least one of: resource usage and input or output data.