Data mining and model generation using an in-database analytic flow generator
View Patent ↗Embodiments are described for a system and method of providing a data miner that decouples the analytic flow solution components from the data source. An analytic-flow solution then couples with the target data source through a simple set of data source connector, table and transformation objects, to perform the requisite analytic flow function. As a result, the analytic-flow solution needs to be designed only once and can be re-used across multiple target data sources. The analytic flow can be modified and updated at one place and then deployed for use on various different target data sources.
1. A method of generating a predictive model to be executed on data stored in a data store and managed by a database management system, comprising:
defining an analytic flow on data within a data source, the analytic flow comprising a plurality of operations to be executed on the data stored in the data store, the analytic flow comprising predictive model steps;
decoupling the analytic flow from the data source;
coupling the analytic flow with the data store, the coupling comprising pushing the analytic flow to the data store such that the analytic flow operates directly on the data in the data store;
executing the analytic flow on the data stored in the data store using the database management system to generate a scored data set for objects defined by the data, wherein the scored data set includes the data stored in the data store and an indication of a presence and degree of error in the scored data set;
appending prediction attributes and confidence measures to the scored data set; and
storing the prediction attributes and confidence measures as part of the data in the data store.
2. The method of claim 1 wherein the analytic flow comprises a plurality of operators, and wherein each operator is linked to a specific individual database management operation.
3. The method of claim 2 wherein each operator is associated with at least one preceding operator and at least one succeeding operator, and receives data from the at least one preceding operator and transmits data to the at least one succeeding operator.
4. The method of claim 3 further comprising validating the analytic flow by verifying that each operator is defined and has appropriate parameters associated therewith, and that each of the at least one preceding operator and succeeding operator exists for each operator.
5. The method of claim 1 further comprising preparing data in the data source for execution of the analytic flow, by performing data conditioning steps that may include at least one of normalizing the data, aggregating the data, and averaging the data.
6. The method of claim 5 wherein the predictive model steps are performed using predictive algorithms selected from the group consisting of: logistic regression, CART tree, decision tree, Naïve Bayes, and Neural network algorithms.
7. The method of claim 1 further comprising, providing a graphical user interface for designing and defining the analytic flow, the graphical user interface providing icons representing nodes in the analytic flow, wherein each node encapsulates a pointer to a physical location in which the data is stored in the data store, and an operator for transformation or algorithmic processing of the data.
8. The method of claim 7 wherein the graphical user interface further comprises output processes to display messages associated with the execution of each node as the analytic flow is performed on the data.
9. The method of claim 1 wherein the analytic flow defines a predictive model, and wherein the method further comprises packaging the predictive model as a web-executable component for sharing and execution in a networked computing environment.
10. A system for generating a predictive model to be executed on data stored in a data store and managed by a database management system, comprising:
a data miner component for defining an analytic flow on data within a data source, the analytic flow comprising a plurality of operations to be executed on the data stored in the data store, the analytic flow comprising predictive model steps, the data miner component also being for decoupling the analytic flow from the data source, and coupling the analytic flow with the data store, the coupling comprising pushing the analytic flow to the data store such that the analytic flow operates directly on the data in the data store;
a database management system component executing the analytic flow on the data stored in the data store to generate a scored data set for objects defined by the data, wherein the scored data set includes the data stored in the data store and an indication of a presence and degree of error in the scored data set; and
a processing component appending prediction attributes and confidence measures to the scored data set and storing the prediction attributes and confidence measures as part of the data in the data store.
11. The system of claim 10 wherein the analytic flow comprises a plurality of operators, and wherein each operator is linked to a specific individual database management operation.
12. The system of claim 11 wherein each operator is associated with at least one preceding operator and at least one succeeding operator, and receives data from the at least one preceding operator and transmits data to the at least one succeeding operator.
13. The system of claim 12 further comprising a validation component validating the analytic flow by verifying that each operator is defined and has appropriate parameters associated therewith, and that each of the at least one preceding operator and succeeding operator exists for each operator.
14. The system of claim 13 further comprising a preparation component preparing data in the data source for execution of the analytic flow by performing data conditioning steps that may include at least one of normalizing the data, aggregating the data, and averaging the data.
15. The system of claim 14 wherein the predictive model steps are performed using predictive algorithms selected from the group consisting of: logistic regression, CART tree, decision tree, Naïve Bayes, and Neural network algorithms.
16. The system of claim 10 wherein the data miner component includes a graphical user interface component that provides icons representing nodes in the analytic flow, wherein each node encapsulates a pointer to a physical location in which the data is stored in the data store, and an operator for transformation or algorithmic processing of the data.
17. The system of claim 16 further comprising an graphical user output component displaying messages associated with the execution of each node as the analytic flow is performed on the data.
18. The system of claim 10 wherein the analytic flow defines a predictive model, and wherein the method further comprises packaging the predictive model as a web-executable component for sharing and execution in a networked computing environment.
19. A computer program product, comprising a non-transitory machine-readable medium storing instructions, which when executed by a processor, cause a computer to perform a method for generating a predictive model to be executed on data stored in a data store and managed by a database management system, comprising:
defining an analytic flow on data within a data source, the analytic flow comprising a plurality of operations to be executed on the data stored in the data store, the analytic flow comprising predictive model steps;
decoupling the analytic flow from the data source;
coupling the analytic flow with the data store, the coupling comprising pushing the analytic flow to the data store such that the analytic flow operates directly on the data in the data store;
executing the analytic flow on the data stored in the data store using the database management system to generate a scored data set for objects defined by the data, wherein the scored data set includes the data stored in the data store and an indication of a presence and degree of error in the scored data set;
appending prediction attributes and confidence measures to the scored data set; and
storing the prediction attributes and confidence measures as part of the data in the data store.
20. The computer program product of claim 19 wherein the analytic flow within the data store defines a predictive model, the method further comprising packaging the predictive model as a web-executable component for sharing and execution in a networked computing environment.