Process control tool for processing big and wide data
A process control tool for processing wide data from automated manufacturing operations. The tool including a feature selector, an analysis server, and a visualization engine. The feature selector receives process input data from at least one manufacturing process application, wherein the process input data includes a plurality of observations and associated variables, converts the received process input data to a stacked format having one row for each variable in each observation, converts identified categorical variables into numerical variables and identified time-series data into fixed numbers of intervals, computes statistics that measure the strengths of relationships between predictor values and an outcome variable, orders, filters, and pivots the predictor values. The analysis server performs at least one operation to identify interactions between predictor values, e.g. using maximum Likelihood computations or predefined searches, in the filtered predictor values. The visualization engine displays the interactions for use in managing the manufacturing operations.
1. A process control tool for processing wide data from automated manufacturing operations, the process control tool comprising:
a feature selector configured to:
receive process input data from at least one manufacturing process application, wherein the process input data includes a plurality of observations and associated variables;
convert identified categorical variables into numerical variables and identified time-series data into fixed numbers of intervals; and
an analysis server configured to:
perform at least one operation to identify interactions between predictor values;
receive a subset of the predictor values; and
perform at least one of a statistical and modeling operation and pre-defined searches to identify interactions between the predictor values in the subset; and
a visualization engine configured to display the interactions for use in managing the manufacturing operations;
wherein the categorical variables are converted into the numerical variables by randomly dividing the categorical variables into two random samples, separately and individually computing an average for each category in each categorical variable, and replacing the categorical variables in each random sample with a computed average for the categorical variables in the other random sample;
wherein the categorial variables are discrete variables and the numerical variables are continuous variables.
2. The process control tool of claim 1 wherein the feature selector is further configured to convert the received process input data to a stacked format having one row for each variable in each observation.
3. The process control tool of claim 1 wherein the identified categorical variables are converted into numerical variables using an impact-coding technique.
4. The process control tool of claim 2 wherein the feature selector is further configured to:
order the predictor values based on the computed statistics;
filter the predictor values to the subset of predictor values based on a threshold value; and
pivot the subset into a wide format wherein each variable is expressed with its own column in a tabular dataset.
5. The process control tool of claim 4 ,
wherein the statistical and modeling operations includes at least one of regression algorithms, neural networks, deep-learning networks, and recursive partitioning algorithms.
6. The process control tool of claim 1 wherein the analysis server is further configured to operate in at least one of in-memory, in-virtual-memory, and multithreaded computations.
7. The process control tool of claim 1 wherein the feature selector and the analysis server work concurrently and asynchronously.
8. The process control tool of claim 1 wherein the analysis server is a dedicated analysis server.
9. The process control tool of claim 1 wherein the process input data is characterized as having a wide data set or a wide and big data set.
10. A method of processing wide data from automated manufacturing operations, the method comprising:
receive process input data from at least one manufacturing process application, wherein the process input data includes a plurality of observations and associated variables;
converting identified categorical variables into numerical variables and identified time-series data into fixed numbers of intervals;
computing statistics that measure the strengths of relationships between predictor values and an outcome variable;
performing at least one operation to identify interactions between predictor values;
receiving a subset of the predictor values;
performing at least one of a statistical and modeling operation and pre-defined searches to identify interactions between the predictor values in the subset; and
displaying the interactions for use in managing the manufacturing operations;
wherein converting the categorical variables into the numerical variables comprises:
randomly dividing the categorical variables into two random samples;
separately and individually computing an average for each category in each categorical variable; and
replacing the categorical variables in each random sample with a computed average for the categorical variables in the other random sample;
wherein the categorial variables are discrete variables and the numerical variables are continuous variables.
11. The method of claim 10 further comprising converting the received process input data to a stacked format having one row for each variable in each observation.
12. The method of claim 10 further comprises converting the identified categorical variables into numerical variables using an impact-coding technique.
13. The method of claim 11 further comprising:
ordering the predictor values based on the computed statistics;
filtering the predictor values to the subset of predictor values based on a threshold value; and
pivoting the subset into a wide format.
14. The method of claim 13 further comprising:
performing at least one of maximum Likelihood computations and pre-defined searches to identify interactions between the predictor values in the subset.
15. A non-transitory computer readable storage medium comprising a set of computer instructions executable by a processor for processing wide data from automated manufacturing operations, the computer instructions configured to:
receive process input data from at least one manufacturing process application, wherein the process input data includes a plurality of observations and associated variables;
convert identified categorical variables into numerical variables and identified time-series data into fixed numbers of intervals;
compute statistics that measure the strengths of relationships between predictor values and an outcome variable;
perform at least one operation to identify interactions between predictor values;
receive a subset of predictor values; and
perform at least one of a statistical and modeling operation and pre-defined searches to identify interactions between the predictor values in the subset; and
display the interactions for use in managing the manufacturing operations;
wherein the categorical variables are converted into the numerical variables by randomly dividing the categorical variables into two random samples, separately and individually computing an average for each category in each categorical variable, and replacing the categorical variables in each random sample with a computed average for the categorical variables in the other random sample;
wherein the categorial variables are discrete variables and the numerical variables are continuous variables.
16. The non-transitory computer readable storage medium as recited in claim 15 further including computer instructions configured to convert the received process input data to a stacked format having one row for each variable in each observation.
17. The non-transitory computer readable storage medium as recited in claim 15 further including computer instructions configured to use an impact-coding technique to convert the identified categorical variables into numerical variables.
18. The non-transitory computer readable storage medium as recited in claim 16 further including computer instructions configured to:
order the predictor values based on the computed statistics;
filter the predictor values to the subset of predictor values based on a threshold value;
pivot the subset into a wide format; and
perform at least one of maximum Likelihood computations and pre-defined searches to identify interactions between the predictor values in the subset.
19. The non-transitory computer readable storage medium as recited in claim 15 further including computer instructions configured wherein the analysis server is further configured to perform at least one operation to identify interactions between predictor values and the at least one outcome variable in at least one of in-memory, in-virtual-memory, and multithreaded computations.
20. The non-transitory computer readable storage medium as recited in claim 15 wherein a subset of the instructions operate concurrently with and asynchronously from another subset of the instructions.