System for rapid interactive exploration of big data
An apparatus for estimating analytics and interactive exploration of big data, stored and/or streaming, using approximate query processing is presented. The apparatus comprises a model constructor and a sampler. The model constructor identifies important predictors variables in big data using feature selection, predictor variables, and outcome variables and partitions the important predictor variables into one or more stratifications based either the identified interactions or identified relationships. The sampler generates a subset of data by querying the big data using a query constructed based on at least one stratification. The subset of data can be fed into an analytics generator. The analytics generator generates analytics data for the outcome variables based on the subset of data and an analytics algorithm and a visualization, e.g. an interactive visualization, comprising the outcome variables, the important predictor variables, the stratification, the subset of data, and the analytics data.
1. A system for estimating analytics and interactive exploration of big data, the system comprising:
a model constructor configured to:
identify in the big data predictors variables using feature selection, at least one predictor variable, and at least one outcome variable;
identify in the big data at least one of interactions between the predictor variables and relationships of structure in the predictor variables using an incremental machine learning algorithm until a desired level of structure has been identified based on a user-defined criteria, wherein the incremental machine learning is used to update the predictor variables based on newly obtained data without a need to reprocess over all previously processed data, wherein the model constructor is further configured to identify a contribution of one predictor variable of the predictor variables as modified by one or more other predictor variables of the predictor variables to identify an interaction of the at least one interactions between the predicator variables, wherein a combined contribution of all predicator variables involved in the interaction is greater than a simple sum over individual contributions of each of the predictor variables;
interaction means that the contribution of one predictor variable is modified by one or many other predictor variables, so that the combined contribution of all variables involved in the interaction is greater than the simple sum over the individual contributions attributable to each variable;
partition the predictor variables into at least one stratification based on at least one of the identified interactions and identified relationships; and
a sampler configured to:
generate a subset of data by querying the big data using a query constructed based on the at least one stratification;
an analytics generator configured to:
generate analytics data for the at least one outcome variable based on the subset of data and at least one analytics algorithm; and
generate at least one visualization comprising at least one of the at least one outcome variable, the predictor variables, the at least one stratification, the subset of data, and the analytics data;
wherein the big data is at least one of stored data and streaming data.
2. The system of claim 1 , wherein the at least one predictor variable and the at least one outcome variable are update variables and the subset of data is an updated subset of data.
3. The system of claim 1 , wherein the predictor variables are identified and partitioned using at least one tree algorithm; wherein the query is constructed using approximate query processing.
4. The system of claim 1 , wherein the predictor variables and outcome variables are at least one of continuous and categorical variables of interest selected from logical data columns.
5. The system of claim 1 , wherein the sampler is further configured to generate the subset of data by querying the big data using a query constructed based on the at least one stratification according to an accuracy criteria.
6. The system of claim 5 , wherein the accuracy criteria is one of time and an acceptable error limit.
7. An apparatus for estimating analytics and interactive exploration of big data, the apparatus comprising:
a model constructor configured to:
continuously identify in the big data at least one of interactions between the predictor variables and relationships of structure in the predictor variables using an incremental machine learning algorithm until a desired level of structure has been identified based on a user-defined criteria, wherein the incremental machine learning is used to update the predictor variables based on newly obtained data without a need to reprocess over all previously processed data,
wherein the model constructor is further configured to identify a contribution of one predictor variable of the predictor variables as modified by one or more other predictor variables of the predictor variables to identify an interaction of the at least one interactions between the predicator variables, wherein a combined contribution of all predicator variables involved in the interaction is greater than a simple sum over individual contributions of each of the predictor variables;
partition, continuously, predictor variables into at least one stratification based on at least one of the identified interactions and identified relationships; and
a sampler configured to:
continuously generate a subset of data by querying the big data using a query constructed based on the at least one stratification;
wherein the big data is at least one of stored data and streaming data.
8. The apparatus of claim 7 , wherein the model constructor is further configured to:
identify in the big data predictors variables using feature selection, at least one predictor variable, and at least one outcome variable.
9. The apparatus of claim 7 , further comprising:
an analytics generator configured to:
generate analytics data for the at least one outcome variable based on the subset of data and at least one analytics algorithm; and
generate at least one visualization comprising at least one of the at least one outcome variable, the predictor variables, the at least one stratification, the subset of data, and the analytics data.
10. The apparatus of claim 7 , wherein the at least one predictor variable and the at least one outcome variable are update variables and the subset of data is an updated subset of data.
11. The apparatus of claim 7 , wherein the predictor variables are identified and partitioned using at least one tree algorithm; wherein the query is constructed using approximate query processing.
12. The apparatus of claim 7 , wherein the predictor variables and outcome variables are at least one of continuous and categorical variables of interest selected from logical data columns.
13. The apparatus of claim 7 , wherein the sampler is further configured to generate the subset of data by querying the big data using a query constructed based on the at least one stratification according to an accuracy criteria.
14. The system of claim 12 , wherein the accuracy criteria is one of time and an acceptable error limit.
15. A method for estimating analytics and interactive exploration of big data, the method comprising:
identify in the big data predictors variables using feature selection, at least one predictor variable, and at least one outcome variable;
identify in the big data at least one of interactions between the predictor variables and relationships of structure in the predictor variables using an incremental machine learning algorithm until a desired level of structure has been identified based on a user-defined criteria, wherein the incremental machine learning is used to update the predictor variables based on newly obtained data without a need to reprocess over all previously processed data, wherein identifying an interaction of the at least one interactions comprises identifying a contribution of one predictor variable of the predictor variables as modified by one or more other predictor variables of the predictor variables, wherein a combined contribution of all predicator variables involved in the interaction is greater than a simple sum over individual contributions of each of the predictor variables;
partition the predictor variables into at least one stratification based on at least one of the identified interactions and identified relationships;
generate a subset of data by querying the big data using a query constructed based on the at least one stratification;
generate analytics data for the at least one outcome variable based on the subset of data and at least one analytics algorithm; and
generate at least one visualization comprising at least one of the at least one outcome variable, the predictor variables, the at least one stratification, the subset of data, and the analytics data;
wherein the big data is at least one of stored data and streaming data.
16. The method of claim 15 , wherein the at least one predictor variable and the at least one outcome variable are update variables and the subset of data is an updated subset of data.
17. The method of claim 15 , wherein the predictor variables are identified and partitioned using at least one tree algorithm; wherein the query is constructed using approximate query processing.
18. The method of claim 15 , further comprises:
generating the subset of data by querying the big data using a query constructed based on the at least one stratification according to an accuracy criteria;
wherein the accuracy criteria is one of time and an acceptable error limit;
wherein the predictor variables and outcome variables are at least one of continuous and categorical variables of interest selected from logical data columns.
19. A system for estimating analytics and interactive exploration of big data, the system comprising:
a storage medium; and
one or more processors configured to:
identify in the big data predictors variables using feature selection, at least one predictor variable, and at least one outcome variable;
identify in the big data at least one of interactions between the predictor variables and relationships of structure in the predictor variables using an incremental machine learning algorithm until a desired level of structure has been identified based on a user-defined criteria, wherein the incremental machine learning is used to update the predictor variables based on newly obtained data without a need to reprocess over all previously processed data, wherein identifying an interaction of the at least one interactions comprises identifying a contribution of one predictor variable of the predictor variables as modified by one or more other predictor variables of the predictor variables, wherein a combined contribution of all predicator variables involved in the interaction is greater than a simple sum over individual contributions of each of the predictor variables;
partition the predictor variables into at least one stratification based on at least one of the identified interactions and identified relationships;
generate a subset of data by querying the big data using a query constructed based on the at least one stratification;
generate analytics data for the at least one outcome variable based on the subset of data and at least one analytics algorithm; and
generate at least one visualization comprising at least one of the at least one outcome variable, the predictor variables, the at least one stratification, the subset of data, and the analytics data;
wherein the big data is at least one of stored data and streaming data.