Performing data mining operations within a columnar database management system
Data mining operations are performed within a columnar database management system. The columnar database management system stores input sets of data for a data mining operation. An input set of data is represented as a column of data in the columnar database management system. The columnar database management system stores instructions to perform one or more data mining operations for processing the input sets of data. The columnar database management system receives requests for performing data mining operations and performs the processing of the data mining operation within the columnar database management system. As a result, the processing of data mining operations is performed without requiring multiple data transfers between an application implementing the data mining operations and the columnar database management system.
1. A method of performing data mining operations in a database system, the method comprising:
receiving, by a data mining system from a client device over a network, a request for a data mining operation, the data mining system having a columnar database management system, the columnar database management system having a database engine;
parsing, by the database engine, the request for the data mining operation, the request specifying data items for the data mining operation;
determining, by the database engine based on the parsing, an internal data mining operation necessary for processing the request for the data mining operation, the internal data mining operation not specified by the request for the data mining operation;
determining, by the database engine, an order in which the data items are to be processed using the data mining operation specified by the request and the internal data mining operation not specified by the request;
generating, by the database engine based at least in part on the order in which the data items are to be processed, a data mining processing plan, the data mining processing plan including the data mining operation specified by the request and the internal data mining operation not specified by the request, the internal data mining operation implemented as a database entity within the columnar database management system;
executing, by the database engine, the data mining processing plan, the executing comprising performing the internal data mining operation and the data mining operation in accordance with the order, wherein the internal data mining operation generates an output or result and wherein the data mining operation uses the output or result from the internal data mining operation as input and generates a data mining result, wherein the executing further comprises:
calculating a normalized information gain for each respective attribute of a set of attributes specified by a data item of the data items by using a cross tabulation entity or a statistical entity depending on data type and distribution;
from the set of attributes specified by the data item, selecting a best attribute based on the normalized information gain calculated by the database engine; and
creating a decision node for a decision tree, the decision node representing the best attribute; and
returning, by the data mining system to the client device over the network, the data mining result for presentation on the client device.
2. The method according to claim 1 , wherein the database engine is adapted for processing data stored by the columnar database management system and for performing data mining using the data and data mining operations implemented as database entities within the columnar database management system.
3. The method according to claim 1 , further comprising:
retrieving the database entity for the internal data mining operation not specified by the request.
4. The method according to claim 1 , wherein the data mining operation specified by the request comprises a decision tree for classifying the data items and wherein the internal data mining operation not specified by the request comprises a cross tabulation algorithm.
5. The method according to claim 1 , wherein
the creating including creating child nodes.
6. The method according to claim 5 , wherein each child node of the child nodes represents a discrete value of the respective attribute.
7. The method according to claim 5 , further comprising:
for each child node of the child nodes, iteratively performing the calculating, the selecting, and the creating until no attributes are left or until all data items belong to same class.
8. A data mining system, comprising:
a processor;
a non-transitory computer-readable medium; and
stored instructions translatable by the processor for:
receiving, from a client device over a network, a request for a data mining operation, the data mining system further comprising a columnar database management system, the columnar database management system having a database engine;
parsing, by the database engine, the request for the data mining operation, the request specifying data items for the data mining operation;
determining, by the database engine based on the parsing, an internal data mining operation necessary for processing the request for the data mining operation, the internal data mining operation not specified by the request for the data mining operation;
determining, by the database engine, an order in which the data items are to be processed using the data mining operation specified by the request and the internal data mining operation not specified by the request;
generating, by the database engine based at least in part on the order in which the data items are to be processed, a data mining processing plan, the data mining processing plan including the data mining operation specified by the request and the internal data mining operation not specified by the request, the internal data mining operation implemented as a database entity within the columnar database management system;
executing, by the database engine, the data mining processing plan, the executing comprising performing the internal data mining operation and the data mining operation in accordance with the order, wherein the internal data mining operation generates an output or result and wherein the data mining operation uses the output or result from the internal data mining operation as input and generates a data mining result, wherein the executing further comprises:
calculating a normalized information gain for each respective attribute of a set of attributes specified by a data item of the data items by using a cross tabulation entity or a statistical entity depending on data type and distribution;
from the set of attributes specified by the data item, selecting a best attribute based on the normalized information gain calculated by the database engine; and
creating a decision node for a decision tree, the decision node representing the best attribute; and
returning, to the client device over the network, the data mining result for presentation on the client device.
9. The data mining system of claim 8 , wherein the database engine is adapted for processing data stored by the columnar database management system and for performing data mining using the data and data mining operations implemented as database entities within the columnar database management system.
10. The data mining system of claim 8 , wherein the stored instructions are further translatable by the processor for:
retrieving the database entity for the internal data mining operation not specified by the request.
11. The data mining system of claim 8 , wherein the data mining operation specified by the request comprises a decision tree for classifying the data items and wherein the internal data mining operation not specified by the request comprises a cross tabulation algorithm.
12. The data mining system of claim 8 , wherein
the creating including creating child nodes.
13. The data mining system of claim 12 , wherein each child node of the child nodes represents a discrete value of the respective attribute.
14. The data mining system of claim 12 , wherein the stored instructions are further translatable by the processor for:
for each child node of the child nodes, iteratively performing the calculating, the selecting, and the creating until no attributes are left or until all data items belong to same class.
15. A computer program product comprising a non-transitory computer-readable medium storing instructions translatable by a processor of a data mining system for:
receiving, from a client device over a network, a request for a data mining operation, the data mining system comprising a columnar database management system, the columnar database management system having a database engine;
parsing, by the database engine, the request for the data mining operation, the request specifying data items for the data mining operation;
determining, by the database engine based on the parsing, an internal data mining operation necessary for processing the request for the data mining operation, the internal data mining operation not specified by the request for the data mining operation;
determining, by the database engine, an order in which the data items are to be processed using the data mining operation specified by the request and the internal data mining operation not specified by the request;
generating, by the database engine based at least in part on the order in which the data items are to be processed, a data mining processing plan, the data mining processing plan including the data mining operation specified by the request and the internal data mining operation not specified by the request, the internal data mining operation implemented as a database entity within the columnar database management system;
executing, by the database engine, the data mining processing plan, the executing comprising performing the internal data mining operation and the data mining operation in accordance with the order, wherein the internal data mining operation generates an output or result and wherein the data mining operation uses the output or result from the internal data mining operation as input and generates a data mining result, wherein the executing further comprises:
calculating a normalized information gain for each respective attribute of a set of attributes specified by a data item of the data items by using a cross tabulation entity or a statistical entity depending on data type and distribution;
from the set of attributes specified by the data item, selecting a best attribute based on the normalized information gain calculated by the database engine; and
creating a decision node for a decision tree, the decision node representing the best attribute; and
returning, to the client device over the network, the data mining result for presentation on the client device.
16. The computer program product of claim 15 , wherein the database engine is adapted for processing data stored by the columnar database management system and for performing data mining using the data and data mining operations implemented as database entities within the columnar database management system.
17. The computer program product of claim 15 , wherein the instructions are further translatable by the processor for:
retrieving the database entity for the internal data mining operation not specified by the request.
18. The computer program product of claim 15 , wherein the data mining operation specified by the request comprises a decision tree for classifying the data items and wherein the internal data mining operation not specified by the request comprises a cross tabulation algorithm.
19. The computer program product of claim 15 , wherein
the creating including creating child nodes.
20. The computer program product of claim 19 , wherein each child node of the child nodes represents a discrete value of the respective attribute.