Distributed data mining and compression method and system
A distributed data mining method and system includes a mediator and a plurality of agents, each of said plurality of agents having a local database. The mediator invokes the agents and each agent performs an attribute/value selection process. The agents pass their respective best attribute/value pair to the mediator and the mediator determines a winning agent from the submissions. The agents are notified of the winning selection and the winner then begins data splitting based on the willing attribute/value pair. The winning agent forwards a split information index to the mediator. The mediator provides the split information index to other, non-winning agents and the agents generate rules for the data mining.
1. A method for distributed data mining, comprising the steps of:
invoking a plurality of agents by a mediator;
beginning attribute selection by each agent, wherein attribute, selection being the selection of one data attribute from a set of local data attributes unique to the respective agent such that the selected data attribute has the highest information gain value among all local attributes;
collecting the highest information gain values from the plurality of agents by the mediator, wherein the highest information gain value of a respective agent is based on its own local data with its own unique data attributes;
selecting by the mediator of a winning agent, wherein the winning agent is the only agent from the plurality of agents with access to the local data attribute with the highest global information gain value;
initiating data splitting by said winning agent based on the value of the data attribute with the highest information gain wherein the specified data attribute is unique to the respective agent's local data;
forwarding split data index information resulting from said data splitting by said winning agent to said mediator;
forwarding said split data index information from said mediator to each of said plurality of agents;
initiating data splitting by each of said plurality of agents other than said winning agent based on the split data index information furnished by the winning agent and broadcasted by the mediator;
generating and saving partial rules by repeating the attribute selection and data splitting process recursively and by tracking the attribute/split information coming from that iteration's winning agent; and
outputting complete rules obtained at the completion of the mining process to said plurality of agents.
2. A The method as claimed in claim 1 , wherein said plurality of agents include non-winning agents, and further comprising the step of:
obtaining split data index information by said non-winning agents from said mediator.
3. A The method as claimed in claim 1 , wherein said split data index information is compressed.