Adaptive sampling via adaptive optimal experimental designs to extract maximum information from large data repositories
A system, method, and computer-readable medium for extracting the samples from big data to extract most information about the relationships of interest between dimensions and variables in the data repository. More specifically, extracting information from large data repositories follows an adaptive process that uses systematic sampling procedures derived from optimal experimental designs to target from a large data set specific observations with information value of interest for the analytic task under consideration. The application of adaptive optimal design to guide exploration of large data repositories provides advantages over known big data technologies.
1. A computer-implementable method for identifying information within a data repository, comprising:
selecting variables of interest within the data repository to provide selected variables;
defining a depth of interactions of interest with respect to the variables of interest to provide a defined depth of interactions;
applying an optimal experimental design operation to the selected variables and the defined depth of interactions, the applying comprising determining a design matrix based upon the optimal experimental design operation;
selecting, using at least one query, a sample case from the data repository, the at least one query being based on the design matrix, wherein a computational effort associated with selecting the sample case from the data repository is independent of a size of data in the data repository, and is dependent on a prediction model associated with selecting the sample case;
loading the sample case into the design matrix, resulting in a sample matrix of the data repository; and
performing modeling operations based on the sample matrix of the data repository.
2. The method of claim 1 , wherein:
defining the depth of interactions that are of interest further comprises determining whether to consider information that can be extracted using each variable of interest.
3. The method of claim 1 , wherein:
defining the depth of interactions that are of interest further comprises determining whether to consider certain interactions of the variables of interest.
4. The method of claim 3 , wherein:
the certain interactions comprise two-way interactions between two design vectors based upon the variables of interest.
5. The method of claim 1 , wherein:
defining the depth of interactions that are of interest further comprises determining types of variables to consider, the types of variables being identified as continuous variables, rank-ordered variables and discrete variables.
6. The method of claim 1 , wherein:
defining the depth of interactions that are of interest further comprises identifying known constraints on relationships between the variables of interest.
7. The system of claim 1 , wherein:
defining the depth of interactions that are of interest further comprises identifying known constraints on relationships between the variables of interest.
8. The method of claim 1 , wherein the design matrix comprises an optimal design matrix, and wherein the sample case is defined through rows of the design matrix.
9. The method of claim 1 , wherein defining the depth of interactions of interest comprises identifying a range of values associated with the variables of interest and dividing the range of values into categories of interest.
10. The method of claim 1 , wherein defining the depth of interactions of interest comprises identifying a range of values associated with the variables of interest and dividing the range of values based on applying an optimal binning operation to the range of values.
11. A system comprising:
a processor;
a data bus coupled to the processor; and
a non-transitory, computer-readable storage medium embodying computer program code, the non-transitory, computer-readable storage medium being coupled to the data bus, the computer program code interacting with a plurality of computer operations and comprising instructions executable by the processor and configured for: selecting variables of interest within the data repository to provide selected variables;
defining a depth of interactions of interest with respect to the variables of interest to provide a defined depth of interactions;
applying an optimal experimental design operation to the selected variables and the defined depth of interactions, the applying comprising determining a design matrix based upon the optimal experimental design operation;
selecting, using at least one query, a sample case from the data repository, the at least one query being based on the design matrix, wherein a computational effort associated with selecting the sample case from the data repository is independent of a size of data in the data repository, and is dependent on a prediction model associated with selecting the sample case;
loading the sample case into the design matrix, resulting in a sample matrix of the data repository; and
performing modeling operations based on the sample matrix of the data repository.
12. The system of claim 11 , wherein:
defining the depth of interactions that are of interest further comprises determining whether to consider information that can be extracted using each variable of interest.
13. The system of claim 11 , wherein:
defining the depth of interactions that are of interest further comprises determining whether to consider certain interactions of the variables of interest.
14. The system of claim 13 , wherein:
the certain interactions comprise two-way interactions between two design vectors based upon the variables of interest.
15. The system of claim 11 , wherein:
defining the depth of interactions that are of interest further comprises determining types of variables to consider, the types of variables being identified as at least one of continuous variables, rank-ordered variables and discrete variables.
16. A non-transitory, computer-readable storage medium embodying computer program code, the computer program code comprising computer executable instructions configured for:
selecting variables of interest within the data repository to provide selected variables;
defining a depth of interactions of interest with respect to the variables of interest to provide a defined depth of interactions;
applying an optimal experimental design operation to the selected variables and the defined depth of interactions, the applying comprising determining a design matrix based upon the optimal experimental design operation;
selecting, using at least one query, a sample case from the data repository, the at least one query being based on the design matrix, wherein a computational effort associated with selecting the sample case from the data repository is independent of a size of data in the data repository, and is dependent on a selection model associated with selecting the sample case;
loading the sample case into the design matrix, resulting in a sample matrix of the data repository; and
performing modeling operations based on the sample matrix of the data repository.
17. The non-transitory, computer-readable storage medium of claim 16 , wherein:
defining the depth of interactions that are of interest further comprises determining whether to consider information that can be extracted using each variable of interest or consider certain interactions of the variables of interest.
18. The non-transitory, computer-readable storage medium of claim 16 , wherein:
defining the depth of interactions that are of interest further comprises determining types of variables to consider, the types of variables being identified as at least one of continuous variables, rank-ordered variables and discrete variables.
19. The non-transitory, computer-readable storage medium of claim 16 , wherein:
defining the depth of interactions that are of interest further comprises identifying known constraints on relationships between the variables of interest.