Techniques for analyzing streaming data
A system and method include receiving a request to analyze streaming data on a grid computing environment, sending, by a primary control node, a portion of the streaming data to each of the plurality of worker nodes, such that each of the plurality of worker nodes analyzes the portion of the streaming data received from the primary control node, receiving, by the primary control node, sub-results from each of the plurality of worker nodes, combining, by the primary control node, the sub-results to compute a result, and outputting, by the primary control node, the result analyzing the streaming data.
1 . A non-transitory computer-readable medium having computer-readable instructions stored thereon for learning a causal structure from multivariate input data that when executed by a processing circuit causes the processing circuit to:
receive multivariate observation data from a plurality of data sources, the multivariate observation data comprising variable values associated with a plurality of variables;
initialize, by the processing circuit executing a graph-learning module stored in memory, a structural dependency model comprising a directed acyclic graph (DAG) template that defines potential causal relationships among the plurality of variables;
transform, by the processing circuit, the multivariate observation data into a structured dependency representation configured for causal structure analysis, wherein the structured dependency representation captures relationships among the plurality of variables for use in learning causal connections;
execute, by the processing circuit executing the graph-learning module, an iterative structure-learning operation to determine a topological order of the DAG based on the structured dependency representation, wherein each iteration dynamically updates:
(i) an index identifying a candidate variable for edge evaluation,
(ii) a model evaluation score representing a causal consistency metric, and
(iii) an ordered list of variables stored in the memory, and
wherein the updates are applied through memory-level transformations of the structured dependency representation without exposing intermediate computation results externally;
detect convergence of the topological order when the model evaluation score satisfies a predefined accuracy criterion; and
store the learned topological order in memory for implementation in subsequent causal inference operations that utilize the learned causal relationships of the DAG to perform predictive, discovery, estimation, or diagnostic computations.
2 . The non-transitory computer-readable medium of claim 1 , wherein:
the plurality of data sources comprises two or more network devices, and
the topological order of the DAG is used to analyze the multivariate observation data and transform the multivariate observation data to output data.
3 . The non-transitory computer-readable medium of claim 2 , wherein the computer-readable instructions further cause the processing circuit to:
determine a correct set of edges for the DAG;
generate the DAG based on the topological order of the DAG and the correct set of edges; and
analyze the multivariate observation data based on the DAG to generate the output data.
4 . The non-transitory computer-readable medium of claim 2 , wherein the multivariate observation data is based on input streaming or non-streaming data received from the one or more network devices.
5 . A non-transitory computer-readable medium having computer-readable instructions stored thereon for accelerating a machine learning of a causal graph using a single processing circuit that when executed by a processor cause the processor to:
receive, by an analytical compute environment, multivariate observation data from a plurality of distributed data sources, wherein the observation data comprises variable values associated with a plurality of variables representing causal input parameters in a monitored system;
initialize, by the single processing circuit executing a graph-learning module, a structural dependency model for the monitored system, the structural dependency model comprising a directed acyclic graph (DAG) template stored in memory;
generate, by the single processing circuit, a correlation representation of the observation data by transforming the variable values into a computational structure configured for causal structure analysis;
execute, by the single processing circuit executing the graph-learning circuit, an iterative structure-learning operation to determine a topological order of the DAG based on the correlation representation, wherein each iteration dynamically updates:
(i) a dependency index identifying a next candidate node for edge determination,
(ii) a model evaluation score representing a causal fit metric, and
(iii) a variable order list stored in the memory, and
wherein the dynamic updates are applied through memory-mapped transformations of the correlation representation without exposing intermediate computational results externally;
detect, by the analytical compute environment, convergence of the topological order based on the model evaluation score satisfying a predefined performance threshold;
deploy, by the analytical compute environment, the learned DAG model into a real-time causal inference pipeline that executes predictive operations for the monitored system based on the learned topological order; and
update, by the analytical compute environment, internal configuration parameters associated with machine-learning components executed within the real-time causal inference pipeline based on updates to the learned DAG model, wherein the updated configuration parameters cause dynamic optimization of causal prediction operations.
6 . The non-transitory computer-readable medium of claim 5 , wherein:
the plurality of data sources comprises two or more network devices, and
the topological order of the DAG is used to analyze the multivariate observation data and transform the multivariate observation data to output data.
7 . The non-transitory computer-readable medium of claim 6 , wherein the computer-readable instructions further cause the single processing circuit to:
determine a correct set of edges for the DAG;
generate the DAG based on the topological order of the DAG and the correct set of edges; and
analyze the multivariate observation data based on the DAG to generate the output data.
8 . The non-transitory computer-readable medium of claim 6 , wherein the multivariate observation data is based on input streaming or non-streaming data received from the one or more network devices.
9 . A method for learning a causal structure from multivariate input data comprising:
receiving multivariate observation data from a plurality of data sources, the multivariate observation data comprising variable values associated with a plurality of variables;
initializing, by processing circuit executing a graph-learning module stored in memory, a structural dependency model comprising a directed acyclic graph (DAG) template that defines potential causal relationships among the plurality of variables;
transforming, by the processing circuit, the multivariate observation data into a structured dependency representation configured for causal structure analysis, wherein the structured dependency representation captures relationships among the plurality of variables for use in learning causal connections;
executing, by the processing circuit executing the graph-learning module, an iterative structure-learning operation to determine a topological order of the DAG based on the structured dependency representation, wherein each iteration dynamically updates:
(i) an index identifying a candidate variable for edge evaluation,
(ii) a model evaluation score representing a causal consistency metric, and
(iii) an ordered list of variables stored in the memory, and
wherein the updates are applied through memory-level transformations of the structured dependency representation without exposing intermediate computation results externally;
detecting convergence of the topological order when the model evaluation score satisfies a predefined accuracy criterion; and
storing the learned topological order in memory for implementation in subsequent causal inference operations that utilize the learned causal relationships of the DAG to perform predictive, discovery, estimation, or diagnostic computations.
10 . The method of claim 9 , wherein:
the plurality of data sources comprises two or more network devices, and
the topological order of the DAG is used to analyze the multivariate observation data and transform the multivariate observation data to output data.
11 . The method of claim 10 , further comprising:
determining a correct set of edges for the DAG;
generating the DAG based on the topological order of the DAG and the correct set of edges; and
analyzing the multivariate observation data based on the DAG to generate the output data.
12 . The method of claim 10 , wherein the multivariate observation data is based on input streaming or non-streaming data received from the one or more network devices.
13 . A method for learning a causal structure from multivariate input data comprising:
receiving, by an analytical compute environment, multivariate observation data from a plurality of distributed data sources, wherein the observation data comprises variable values associated with a plurality of variables representing causal input parameters in a monitored system;
initializing, by a single processing circuit executing a graph-learning module, a structural dependency model for the monitored system, the structural dependency model comprising a directed acyclic graph (DAG) template stored in memory;
generating, by the single processing circuit, a correlation representation of the observation data by transforming the variable values into a computational structure configured for causal structure analysis;
executing, by the single processing circuit executing the graph-learning circuit, an iterative structure-learning operation to determine a topological order of the DAG based on the correlation representation, wherein each iteration dynamically updates:
(i) a dependency index identifying a next candidate node for edge determination,
(ii) a model evaluation score representing a causal fit metric, and
(iii) a variable order list stored in the memory, and
wherein the dynamic updates are applied through memory-mapped transformations of the correlation representation without exposing intermediate computational results externally;
detecting, by the analytical compute environment, convergence of the topological order based on the model evaluation score satisfying a predefined performance threshold;
deploying, by the analytical compute environment, the learned DAG model into a real-time causal inference pipeline that executes predictive operations for the monitored system based on the learned topological order; and
updating, by the analytical compute environment, internal configuration parameters associated with machine-learning components executed within the real-time causal inference pipeline based on updates to the learned DAG model, wherein the updated configuration parameters cause dynamic optimization of causal prediction operations.
14 . The method of claim 13 , wherein:
the plurality of data sources comprises two or more network devices, and
the topological order of the DAG is used to analyze the multivariate observation data and transform the multivariate observation data to output data.
15 . The method of claim 14 , further comprising:
determining a correct set of edges for the DAG;
generating the DAG based on the topological order of the DAG and the correct set of edges; and
analyzing the multivariate observation data based on the DAG to generate the output data.
16 . The method of claim 14 , wherein the multivariate observation data is based on input streaming or non-streaming data received from the one or more network devices.
17 . A system for learning a causal structure from multivariate input data comprising:
a memory having computer-readable instructions stored thereon; and
a processing circuit that executes the computer-readable instructions to:
receive multivariate observation data from a plurality of data sources, the multivariate observation data comprising variable values associated with a plurality of variables;
initialize, by the processing circuit executing a graph-learning module stored in memory, a structural dependency model comprising a directed acyclic graph (DAG) template that defines potential causal relationships among the plurality of variables;
transform, by the processing circuit, the multivariate observation data into a structured dependency representation configured for causal structure analysis, wherein the structured dependency representation captures relationships among the plurality of variables for use in learning causal connections;
execute, by the processing circuit executing the graph-learning module, an iterative structure-learning operation to determine a topological order of the DAG based on the structured dependency representation, wherein each iteration dynamically updates:
(i) an index identifying a candidate variable for edge evaluation,
(ii) a model evaluation score representing a causal consistency metric, and
(iii) an ordered list of variables stored in the memory, and
wherein the updates are applied through memory-level transformations of the structured dependency representation without exposing intermediate computation results externally;
detect convergence of the topological order when the model evaluation score satisfies a predefined accuracy criterion; and
store the learned topological order in memory for implementation in subsequent causal inference operations that utilize the learned causal relationships of the DAG to perform predictive, discovery, estimation, or diagnostic computations.
18 . The system of claim 17 , wherein:
the plurality of data sources comprises two or more network devices, and
the topological order of the DAG is used to analyze the multivariate observation data and transform the multivariate observation data to output data.
19 . The system of claim 18 , wherein the processing circuit further executes the computer-readable instructions to:
determine a correct set of edges for the DAG;
generate the DAG based on the topological order of the DAG and the correct set of edges; and
analyze the multivariate observation data based on the DAG to generate the output data.
20 . The system of claim 18 , wherein the multivariate observation data is based on input streaming or non-streaming data received from the one or more network devices.