Automatic detection of patterns and inference in a dataset
View Patent ↗Techniques allow automatic identification of statistically significant attribute combinations in a dataset, and provide users with an understanding thereof including starting points for further analysis. Statistically significant combinations may be obtained from large data sets by limiting combinations to four or fewer attributes. The combinations obtained may be ranked to differentiate patterns, e.g. according to factors such as error ratio, decision tree depth, occurrences, and number of attributes. Still further insights may be achieved by ranking attributes according to the number of statistically significant combinations in which they appear. For useful visualization of statistically significant information within the patterns, only those having at least one measure/numeric may analyzed for further insight (e.g. by an outlier algorithm) and presented as output in a chart (e.g. pie, bar) form. The decision tree approach of various embodiments may facilitate ‘What if’ analysis of the data, as well as obtaining the reverse inference.
1. A computer-implemented method comprising:
providing a data set comprising data organized in rows and columns;
generating a decision tree comprising a combination of between 2-4 columns;
evaluating a statistical significance of the combination of columns;
ranking the combination of columns; and
based upon the ranked combination of columns, performing additional analysis selected from:
ranking of columns;
performing a “What if” analysis; and
obtaining a reverse inference.
2. A method as in claim 1 further comprising:
generating an inference based upon a ranked combination of columns comprising at least one numerical measure; and
displaying the inference as a chart comprising the numerical measure.
3. A method as in claim 1 wherein ranking the combination of columns considers a factor selected from: a decision tree depth, an error ratio, a number of occurrences, a number of attributes, and a boosting factor.
4. A method as in claim 1 wherein the combinations of columns include a continuous variable, and the method further comprises discretizing the continuous variable.
5. A method as in claim 1 further comprising creating a READ ONLY procedure in order to generate the decision tree.
6. A non-transitory computer readable storage medium embodying a computer program for performing a method on a processor, said method comprising:
providing a data set comprising data organized in rows and columns;
generating a decision tree comprising a combination of between 2-4 columns;
evaluating a statistical significance of the combination of columns;
ranking the combination of columns; and
based upon the ranked combination of columns, performing additional analysis selected from:
ranking of columns;
performing a “What if” analysis; and
obtaining a reverse inference.
7. A non-transitory computer readable storage medium as in claim 6 wherein the method further comprises:
generating an inference based upon a ranked combination of columns comprising at least one numerical measure; and
displaying the inference as a chart comprising the numerical measure.
8. A non-transitory computer readable storage medium as in claim 6 wherein ranking the combination of columns considers a factor selected from: a decision tree depth, an error ratio, a number of occurrences, a number of attributes, and a boosting factor.
9. A non-transitory computer readable storage medium as in claim 6 wherein the combinations of columns include a continuous variable, and the method further comprises discretizing the continuous variable.
10. A non-transitory computer readable storage medium as in claim 6 wherein the method further comprises creating a READ ONLY procedure in order to generate the decision tree.
11. A computer system comprising:
one or more processors;
a software program, executable on said computer system, the software program:
providing a data set comprising data organized in rows and columns;
generating a decision tree comprising a combination of between 2-4 columns;
evaluating a statistical significance of the combination of columns;
ranking the combination of columns; and
based upon the ranked combination of columns, performing additional analysis selected from:
ranking of columns;
performing a “What if' analysis; and
obtaining a reverse inference.
12. A computer system as in claim 11 wherein the software program:
generates an inference based upon a ranked combination of columns comprising at least one numerical measure; and
displays the inference as a chart comprising the numerical measure.
13. A computer system as in claim 11 wherein ranking the combination of columns considers a factor selected from: a decision tree depth, an error ratio, a number of occurrences, a number of attributes, and a boosting factor.
14. A computer system as in claim 11 wherein the combinations of columns include a continuous variable, and the method further comprises discretizing the continuous variable.
15. A computer system as in claim 11 further comprises creating a READ ONLY procedure in order to generate the decision tree.