Detecting important variables and their interactions in big data
A system, method, and computer-readable medium for performing a variable and interaction detection operation. In certain embodiments, the variable and interaction detection operation detects important variables and as well as interactions among certain important variables in big data. In certain embodiments, the variable and interaction detection operation includes a robust automated process flow (i.e., an analytic recipe) which identifies important variables and their interactions in large data sets.
1. A computer-implementable method for performing in-database operations on at least one data repository having a high dimensional parameter space, comprising:
instantiating an in-database processing operation, the in-database processing operation including at least one database statement for detecting important variables and interactions among select important variables from the high dimensional parameter space;
distributing the at least one database statement across a plurality of nodes of the storage system;
executing the at least one database statement at each of the plurality of nodes of the storage system to identify an important variable from the high dimensional parameter space; and
presenting information regarding the important variable via an important variable user interface;
wherein executing the at least one database statement includes preprocessing data according to a predictive modeling problem type and a clustering problem type;
wherein an amount of information extracted from a sample of data based on sets of variables defines an important variable;
wherein executing the at least one database statement includes selecting a set of predictor variables;
wherein a probability of each predictor variable selected is proportional to an average goal function value.
2. The method of claim 1 , further comprising:
identifying interactions among a plurality of important variables in the high dimensional parameter space.
3. The method of claim 1 , wherein:
the identifying the important variable comprises executing an analytic recipe to identify important variables and interactions of the important variables.
4. The method of claim 3 , wherein:
the analytic recipe combines repeated sampling from large data sets, applies automated binning of all predictor variables, and performs a search operation to identify the important variables.
5. The method of claim 4 , wherein:
the search operation comprises includes a simulated annealing operation, the simulated annealing operation identifying an importance of a number (k) of variables from among an available number (n) variables, where n is greater than or equal to k.
6. The method of claim 1 , further comprising:
selecting a number of predictor variables for extraction and evaluation;
randomly selecting, iteratively, the number of predictor variables; and
computing, iteratively, a goal function value.
7. The method of claim 1 , further comprising: determining improvement of a goal function.
8. A system comprising:
a processor;
a data bus coupled to the processor; and
a non-transitory, computer-readable storage medium embodying computer program code, the non-transitory, computer-readable storage medium being coupled to the data bus, the computer program code interacting with a plurality of computer operations and comprising instructions executable by the processor and configured for:
instantiating an in-database processing operation, the in-database processing operation including at least one database statement for detecting important variables and interactions among select important variables from a high dimensional parameter space from at least one data repository;
distributing the at least one database statement across a plurality of nodes of the storage system;
executing the at least one database statement at each of the plurality of nodes of the storage system to identify an important variable from the high dimensional parameter space;
presenting information regarding the important variable via an important variable user interface;
wherein executing the at least one database statement includes preprocessing data according to a predictive modeling problem type and a clustering problem type;
wherein an amount of information extracted from a sample of data based on sets of variables defines an important variable;
wherein executing the at least one database statement includes selecting a set of predictor variables;
wherein a probability of each predictor variable selected is proportional to an average goal function value.
9. The system of claim 8 , wherein the instructions are further configured for:
identifying interactions among a plurality of important variables in the high dimensional parameter space.
10. The system of claim 8 , wherein:
the identifying the important variable comprises executing an analytic recipe to identify important variables and interactions of the important variables.
11. The system of claim 10 , wherein:
the analytic recipe combines repeated sampling from large data sets, applies automated binning of all predictor variables, and performs a search operation to identify the important variables.
12. The system of claim 11 , wherein:
the search operation comprises includes a simulated annealing operation, the simulated annealing operation identifying an importance of a number (k) of variables from among an available number (n) variables, where n is greater than or equal to k.
13. The system of claim 8 , wherein the instructions are further configured for:
selecting a number of predictor variables for extraction and evaluation;
randomly selecting the number of predictor variables; and
computing a goal function value.
14. The system of claim 8 , further comprising: determining improvement of a goal function.
15. A non-transitory, computer-readable storage medium embodying computer program code, the computer program code comprising computer executable instructions configured for:
instantiating an in-database processing operation, the in-database processing operation including at least one database statement for detecting important variables and interactions among select important variables from a high dimensional parameter space from at least one data repository;
distributing the at least one database statement across a plurality of nodes of the storage system;
executing the at least one database statement at each of the plurality of nodes of the storage system to identify an important variable from the high dimensional parameter space; and
presenting information regarding the important variable via an important variable user interface;
wherein executing the at least one database statement includes preprocessing data according to a predictive modeling problem type and a clustering problem type;
wherein an amount of information extracted from a sample of data based on sets of variables defines an important variable;
wherein executing the at least one database statement includes selecting a set of predictor variables;
wherein a probability of each predictor variable selected is proportional to an average goal function value.
16. The non-transitory, computer-readable storage medium of claim 15 , wherein the instructions are further configured for:
identifying interactions among a plurality of important variables in the high dimensional parameter space.
17. The non-transitory, computer-readable storage medium of claim 15 , wherein:
the identifying the important variable comprises executing an analytic recipe to identify important variables and interactions of the important variables.
18. The non-transitory, computer-readable storage medium of claim 17 , wherein:
the analytic recipe combines repeated sampling from large data sets, applies automated binning of all predictor variables, and performs a search operation to identify the important variables.
19. The non-transitory, computer-readable storage medium of claim 18 , wherein:
the search operation comprises includes a simulated annealing operation, the simulated annealing operation identifying an importance of a number (k) of variables from among an available number (n) variables, where n is greater than or equal to k.
20. The non-transitory, computer-readable storage medium of claim 15 , wherein the instructions are further configured for:
selecting a number of predictor variables for extraction and evaluation;
randomly selecting the number of predictor variables; and
computing a goal function value.