Data mining and model generation using an in-database analytic flow generator
View Patent ↗Embodiments are described for a system and method of providing a data miner that decouples the analytic flow solution components from the data source. An analytic-flow solution then couples with the target data source through a simple set of data source connector, table and transformation objects, to perform the requisite analytic flow function. As a result, the analytic-flow solution needs to be designed only once and can be re-used across multiple target data sources. The analytic flow can be modified and updated at one place and then deployed for use on various different target data sources.
1. A method of defining an analytic flow within a database to be executed on data stored in a physical target data source and managed by a database management system, comprising:
defining a data source connection object that is unique to the physical target data source as compared to a plurality of other data sources, the data source connection object being a data structure comprising a pointer to the physical target data source;
defining a table object that references the data source connection object and interfaces between the physical target data source and the analytic flow within the database, wherein the analytic flow within the database is created using a template data source, wherein the template data source is different from the physical target data source, and the table object stores a mapping between the physical target data source and the template data source, the table object further storing attributes of the physical target data source and a connection name that points to a specific connection between the data source connection object and the physical target data source;
defining a transformation object that maps the attributes of the physical target data source to template data attributes that are stored in the template data source, wherein the transformation object is defined to exist separate from the table object; and
receiving the analytic flow by the database management system, the analytic flow operating within the database management system on data stored in the physical target data source that has been retrieved using the table object and transformed using the transformation object.
2. The method of claim 1 wherein the physical target data source comprises one of a relational database, and one or more data files.
3. The method of claim 2 wherein the data source connection object comprises a name of the data source connection object and a storage for metadata of the physical target data source.
4. The method of claim 3 wherein the table object further stores one or more attribute names and attribute types for the data source connection object.
5. The method of claim 4 wherein the transformation object stores one or more attributes for the template data source and one or more attributes for the physical target data source, and a mapping relationship between data in the physical target data source and corresponding data in the template data source.
6. The method of claim 1 further comprising, providing a graphical user interface for designing and defining the analytic flow, the graphical user interface providing icons representing nodes in the analytic flow, wherein each node encapsulates a pointer to a physical location in which the data is stored in the data store, and an operator for transformation or algorithmic processing of the data.
7. The method of claim 6 wherein the graphical user interface further comprises output processes to display messages associated with the execution of each node as the analytic flow is performed on the data.
8. The method of claim 1 wherein the analytic flow defines a predictive model comprising predictive algorithms selected from the group consisting of: logistic regression, CART tree, decision tree, Naïve Bayes, and Neural network algorithms.
9. A system for defining an analytic flow within a database to define a predictive model, comprising:
a database managing component managing data for an organization;
a physical target data source storing the data;
a data source connector object that is unique to the physical target data source as compared to a plurality of other data sources, the data source connection object being a data structure comprising a pointer to the physical target data source;
a table object that references the data source connector object and interfaces between the physical target data source and the analytic flow within the database, wherein the analytic flow is created using a template data source, wherein the template data source is different from the physical target data source, and the table object is defined to store a mapping between the physical target data source and the template data source, the table object further storing attributes of the physical target data source and a connection name that points to a specific connection between the data source connection object and the physical target data source;
a template data source storing attributes for the analytic flow within the database; and
a transformation object that maps the attributes of the physical target data source to the template data attributes that are stored in the template data source, wherein the transformation object is defined to exist separate from the table object, wherein the analytic flow is received by the database managing component, the analytic flow operating within the database management system on data stored in the physical target data source that has been retrieved using the table object and transformed using the transformation object.
10. The system of claim 9 wherein the data source connection object comprise a name of the data source connection object and a storage for metadata of the physical target data source.
11. The system of claim 10 wherein the table object further stores one or more attribute names and attribute types for the data source connection object.
12. The system of claim 11 wherein the transformation object stores one or more attributes for the template data source and one or more attributes for the physical target data source, and a mapping relationship between data in the physical target data source and corresponding data in the template data source.
13. A computer program product, comprising a non-transitory machine-readable medium storing instructions, which when executed by a processor, cause a computer to perform a method for defining an analytic flow within a database to be executed on data stored in a physical target data source and managed by a database management system, the method comprising:
defining a data source connection object that is unique to the physical target data source as compared to a plurality of other data sources, the data source connection object being a data structure comprising a pointer to the physical target data source;
defining a table object that references the data source connection object and interfaces between the physical target data source and the analytic flow within the database, the table object storing attributes of the physical target data source and a connection name that points to a specific connection between the data source connection object and the physical target data source;
defining a transformation object that maps the attributes of the physical target data source to template data attributes from which the analytic flow within the database is designed; and
receiving the analytic flow by the database management system, the analytic flow operating within the database management system on data stored in the physical target data source that has been retrieved using the table object and transformed using the transformation object.
14. The computer program product of claim 13 wherein the physical target data source comprises one of a relational database, and one or more data files.
15. The computer program product of claim 14 wherein of the data source connection object comprise a name of the data source connection object, and a storage for metadata of the physical target data source.
16. The computer program product of claim 15 wherein the table object further stores one or more attribute names and attribute types for the data source connection object.
17. The computer program product of claim 16 wherein the transformation object stores one or more attributes for the template data source and one or more attributes for the physical target data source, and a mapping relationship between data in the physical target data source and corresponding data in the template data source.
18. The computer program product of claim 15 further comprising providing a graphical user interface for designing and defining the analytic flow, the graphical user interface providing icons representing nodes in the analytic flow, wherein each node encapsulates a pointer to a physical location in which the data is stored in the data store, and an operator for transformation or algorithmic processing of the data.