ALGORITHMIC LEARNING ENGINE FOR DYNAMICALLY GENERATING PREDICTIVE ANALYTICS FROM HIGH VOLUME, HIGH VELOCITY STREAMING DATA
An algorithmic real-time learning engine comprising an algorithmic model generator configured to process a set of system variables from a big data source using at least one of a pattern recognition algorithm and a statistical test algorithm to identify patterns, relationships between variables, and important variables; and generate at least one of: a predictive model based on the identified patterns, relationships between variables, and important variables; statistical test model about correlations, differences between variables, or patterns in time across variables; and recurring clusters model of similar observations across variables. A data preprocessor can select system variables of interest, align the selected system variables based on time, and arrange the aligned variables into rows. The selected system variables can also be aggregated based on a pre-defined aggregate. A visualization processor generates visualizations based on the set of system variables and the predictive model, the statistical test, or recurring cluster.
1 . An algorithmic learning engine for processing high volume, high velocity streaming data received from a system process, the algorithmic learning engine comprising:
an algorithmic model generator configured to:
process a set of system variables from the streaming data using at least one of a pattern recognition algorithm and a statistical test algorithm to identify patterns, relationships between variables, and important variables; and
generate at least one of:
a predictive model based on the identified patterns, relationships between variables, and important variables;
statistical test model about correlations, differences between variables, or patterns in time across variables; and
recurring clusters model of similar observations across variables.
2 . The algorithmic learning engine of claim 1 , further comprising a data preprocessor configured to select system variables of interest and perform at least one of:
aggregate the selected system variables; and aligning the selected system variables.
3 . The algorithmic learning engine of claim 2 , wherein the data preprocessor is further configured to:
align the selected system variables based on time; and
arrange the aligned variables into rows.
4 . The algorithmic learning engine of claim 3 , wherein the data preprocessor is further configured to aggregate the selected system variables based on at least one pre-defined aggregate.
5 . The algorithmic learning engine of claim 4 , wherein the pre-defined aggregate is at least one of: an average, a maximum value, a minimum value, a maximum value, medians standard deviations.
6 . The algorithmic learning engine of claim 3 , wherein:
the data pre-processor is further configured to augment the logical rows with predictions derived from historical information; and
the algorithmic learning algorithm is further configured to:
generate, incrementally, at least one of:
the predictive model based on the identified patterns, relationships between variables, and important variables;
the statistical test model about correlations, differences between variables, or patterns in time across variables; and
the recurring clusters model of similar observations across variables.
7 . The algorithmic learning engine of claim 1 , further comprising a visualization processor configured to:
generate at least one of a graph, statistical information, and alarm based on the set of system variables and at least one of: the predictive model, the statistical test, and recurring cluster.
8 . A method for processing high volume, high velocity streaming data received from a system process, the method comprising:
processing a set of system variables from the streaming data using at least one of a pattern recognition algorithm and a statistical test algorithm to identify patterns, relationships between variables, and important variables; and
generating at least one of:
a predictive model based on the identified patterns, relationships between variables, and important variables;
a statistical test model about correlations, differences between variables, or patterns in time across variables; and
a recurring clusters model of similar observations across variables.
9 . The method of claim 8 , further comprising:
selecting system variables of interest and perform at least one of:
aggregating the selected system variables; and aligning the selected system variables.
10 . The method of claim 9 , further comprising:
aligning the selected system variables based on time; and arranging the aligned variables into rows.
11 . The method of claim 10 , further comprises aggregating the selected system variables based on at least one pre-defined aggregate.
12 . The method of claim 11 , wherein the pre-defined aggregate is at least one of: an average, a maximum value, a minimum value, a maximum value, medians standard deviations.
13 . The method of claim 11 , further comprising:
augmenting the logical rows with predictions derived from historical information;
generating, incrementally, at least one of: the predictive model based on the identified patterns, relationships between variables, and important variables; the statistical test model about correlations, differences between variables, or patterns in time across variables; and the recurring clusters model of similar observations across variables.
14 . The method of claim 8 , further comprising generating at least one of a graph, statistical information, and alarm based on the set of system variables and at least one of:
the predictive model, the statistical test, and recurring cluster.
15 . A system for processing high volume, high velocity streaming data received from a system process, the system comprising:
a plurality of system process servers configured to:
generate the streaming high volume, high velocity data;
a data preprocessor configured to:
create the set of system variables by performing at least one of aggregating select system variables and aligning select system variables;
an algorithmic model generator configured to:
process a set of system variables from the streaming data using at least one of a pattern recognition algorithm and a statistical test algorithm to identify patterns, relationships between variables, and important variables; and
generate at least one of:
a predictive model based on the identified patterns, relationships between variables, and important variables;
a statistical test model about correlations, differences between variables, or patterns in time across variables; and
a recurring clusters model of similar observations across variables.
16 . The system of claim 15 , wherein the data preprocessor is further configured to:
align the selected system variables based on time; and arrange the aligned variables into rows.
17 . The system of claim 16 , wherein the data preprocessor is further configured to aggregate the selected system variables based on at least one pre-defined aggregate.
18 . The system of claim 17 , wherein the pre-defined aggregate is at least one of: an average, a maximum value, a minimum value, a maximum value, medians standard deviations.
19 . The system of claim 16 , wherein:
the data pre-processor is further configured to: augment the logical rows with predictions derived from historical information; and
the algorithmic model generator is further configured to generate, incrementally, at least one of: the predictive model based on the identified patterns, relationships between variables, and important variables; the statistical test model about correlations, differences between variables, or patterns in time across variables; and the recurring clusters model of similar observations across variables.
20 . The system of claim 15 , further comprising a visualization processor configured to:
generate at least one of a graph, statistical information, and alarm based on the set of system variables and at least one of: the predictive model, the statistical test, and recurring cluster.