DATA PREPARATION FOR DATA MINING
A system for preparing data for data mining can be utilized to automate translation of raw data to denormalized high-dimensional data in a format of vectors by processing the raw data in a computer cluster processing system. In embodiments, a system for preparing data for data mining includes a data assemble definition interface, a data assemble plan generator, a data assemble plan compiler, a cluster execution module, and a data warehouse module. A user may input a data schema that specifies the raw data input, feature extraction or data translate method, output attributes, and output layer attributes. Embodiments of the present disclosure can interpret the data schema, plan a large data processing work flow for a computer cluster, execute the computer cluster process, and output the data in the format specified by the user in the data schema.
1 . A computer-implemented method for preparing data for data mining, comprising:
retrieving raw data pages, wherein the raw data pages each have at least one attribute;
receiving a data schema defining an output data format and one or more output attributes;
at a data assemble plan generator, generating a data assemble plan for the one or more output attributes;
at a data assemble plan compiler, formulating a data pipeline definition according to the data assemble plan;
executing a computer cluster processing operation to process the data according to the data schema; and
at a data warehouse importer, storing the results of the computer cluster processing operation at a data warehouse.
2 . The method claim 1 , wherein the raw data comprises raw structured data.
3 . The method claim 1 , wherein formulating the data pipeline definition comprises creating a Cascading data processing workflow.
4 . The method claim 1 , wherein executing the computer cluster processing operation further comprises implementing a Hadoop MapReduce job.
5 . The method claim 1 , wherein the raw data comprises pages connected by links.
6 . The method claim 5 , wherein the raw data further comprises page attributes describing the pages and link attributes describing the links.
7 . The method claim 6 , wherein the raw data further comprises page attribute attributes describing the page attributes.
8 . The method of claim 1 , wherein the raw data was drawn from a data source selected from the group consisting of web pages, logs of page views, click streams, transaction logs, social network feeds, news feeds, application logs, application server logs, and system logs.
9 . The method of claim 1 , wherein the one or more output attributes comprise selected ones of page attributes, link attributes, and attribute attributes.
10 . A computer-implemented method for preparing data for data mining, comprising:
receiving a user selection that identifies raw data and a desired data output;
generating a data schema for the user selection;
at a data assemble plan generator, interpreting the data schema to create a data assemble plan;
at a data assemble plan compiler, planning a data processing work flow to follow the data assemble plan;
at a computer cluster, processing the raw data according to the data schema; and
at the computer cluster, organizing the raw data according to the data schema.
11 . The method of claim 10 , further comprising storing the data at a data warehouse.
12 . The method of claim 10 , wherein processing the data further comprises featurizing the data.
13 . The method of claim 10 , wherein the raw data comprises raw structured data.
14 . The method of claim 10 , wherein planning a data processing work flow comprises creating a Cascading data processing workflow.
15 . The method of claim 10 , wherein processing the raw data further comprises implementing a Hadoop MapReduce job.
16 . The method of claim 10 , wherein the raw data comprises pages connected by links, page attributes describing the pages, and link attributes describing the links
17 . The method of claim 10 , wherein the raw data was drawn from a data source selected from the group consisting of web pages, logs of page views, click streams, transaction logs, social network feeds, news feeds, application logs, application server logs, and system logs.
18 . The method of claim 10 , wherein the desired data output comprises selected ones of page attributes, link attributes, and attribute attributes.
19 . A computer system for preparing data for data mining comprising:
a data preparation computer device comprising a memory and a processing device, the memory storing computer-readable instructions directing the processing device to:
retrieve raw data pages, wherein the raw data pages each have at least one attribute;
receive a data schema defining an output data format and one or more output attributes;
generate a data assemble plan for the one or more output attributes;
formulate a data pipeline definition according to the data assemble plan;
execute a computer cluster processing operation to process the data according to the data schema and organize the data according to the output data format; and
store the results of the computer cluster processing operation at a data warehouse.
20 . The system of claim 19 , further comprising a Hadoop cluster.