System and method for collecting and processing data
A system and method for collecting and processing data over a communications network. A data mining marshaller module associates each plugin to a particular data source and manages the plugin to periodically retrieve unstructured data from the data source based on a plurality of data items to be monitored on behalf of a plurality of users. The plugins convert unstructured data received from the data sources to structured data and the data marshaller module stores the structured data in a database. This enables the system and method to aggregate and display the structured data in multiple graphical representations according to the user's preference.
1. A system for collecting and processing data over a communications network, comprising:
a processor based web server comprising:
a plurality of plugins stored in the server, each configured to retrieve unstructured data from a data source on behalf of a plurality of users over said communications network and to convert said unstructured data by parsing said unstructured data into structured data comprising historical data, updated data and analytical graphical representation of said data;
an activation module for receiving each user's selection of one or more data items relating to a subject matter to be monitored from a client device associated with said each user over said communications network; and
a data mining marshaller (DMM) module for associating a plugin to each data item selected by said plurality of users, associating said plugin to an associated data source, associating said plugin to each user who have selected same associated data item to be monitored, and managing each plugin to periodically retrieve said unstructured data from said associated data source based on said associated data item to be monitored on behalf of the users who have selected said associated data item to be monitored to enable multiple users to share said structured data based on shared interest in same subject matter; and
a database to store said structured data received from said plurality of plugins.
2. The system of claim 1 , wherein said DMM module of said processor based web server associates said each plugin with one of a plurality of data sources defined by said plurality of users.
3. The system of claim 1 , wherein said each data source is accessed by a plugin associated with said each data source.
4. The system of claim 1 , wherein said DMM module of said processor based web server determines if a data item needs to be updated; and to manage a plugin associated with said data item to update said data item.
5. The system of claim 4 , wherein said DMM module of said processor based web server loads and launches said plugin associated with said data item to retrieve said unstructured data in response to said data item to be monitored.
6. The system of claim 1 , wherein said each plugin further comprises a filter module for filtering said unstructured data in response to a plurality of data item parameters.
7. The system of claim 6 , wherein said filter module is operable to filter said unstructured data in response to one or more data item parameters received from said plurality of users.
8. The system of claim 1 , wherein said DMM module further comprises a filter module for filtering said structured data in response to a plurality of data item parameters.
9. The system of claim 1 , wherein said processor based web server further comprises a display navigation module to display said stored structured data received in response to a data item associated with a user.
10. The system of claim 9 , wherein said display navigation module of said processor based web server displays said stored structured data in a plurality of graphical representations selectable by said user.
11. The system of claim 9 , wherein said display navigation module of said processor based web server displays said structured data in one or more following graphical representations: displaying said structured data related to one data item associated with said user, displaying structured data related to two or more data items associated with said user; displaying structured data associated with said user from a single data source; and displaying structured data associated said user from multiple data sources.
12. The system of claim 11 , wherein said activation module generates a watch group by said user, said watch group comprising one or more data items relating to the subject matter, said associated data sources and said graphical representations of said structured data.
13. The system of claim 12 , wherein said activation module is operable to enable said user to share said watch group with another user.
14. The system of claim 12 , further comprising a processor based user device to access said DMM module of said processor based web server; and wherein said display navigation module of said processor based web server displays said structured data on said processor based user device.
15. A non-transitory computer-readable medium comprising a code for collecting and processing data over a communications network, said code comprising instructions for:
configuring a plurality of plugins stored in processor based server, each plugin configured to retrieve unstructured data from a data source over said communications network and to convert said unstructured data by parsing said unstructured data into structured data comprising historical data, updated data and analytical graphical representation of said data;
receiving each user's selection of one or more data items relating to a subject matter to be monitored from a client device associated with said each user over said communications network;
associating a plugin to each data item selected by said plurality of users;
associating said plugin to an associated data source;
associating said plugin to each user who have selected same associated data item to be monitored;
managing each plugin to periodically retrieve said unstructured data from said associated data source based on said associated data item to be monitored on behalf of the users who have selected said associated data item to be monitored to enable multiple users to share said structured data based on shared interest in same subject matter; and
storing said structured data received from said plurality of plugins in a database.
16. A computer based method for collecting and processing data over a communications network, comprising the steps of:
configuring a plurality of plugins stored in a processor based server, each plugin configured to retrieve unstructured data from a data source over said communications network and to convert said unstructured data by parsing said unstructured data into structured data comprising historical data, updated data and analytical graphical representation of said data;
receiving each user's selection of one or more data items relating to a subject matter to be monitored from a client device associated with said each user over said communications network;
associating a plugin to each data item selected by said plurality of users;
associating said plugin to an associated data source;
associating said plugin to each user who have selected same associated data item to be monitored;
managing each plugin to periodically retrieve said unstructured data from said associated data source based on said associated data item to be monitored on behalf of the users who have selected said associated data item to be monitored to enable multiple users to share said structured data based on shared interest in same subject matter; and
storing said structured data received from said plurality of plugins in a database.
17. The computer based method of claim 16 , wherein said step of associating said each plugin to said data source further comprises associating said each plugin with one of a plurality of data sources defined by said plurality of users.
18. The computer based method of claim 16 , wherein said step of managing said each plugin further comprises managing said each plugin to access a data source.
19. The computer based method of claim 16 , further comprising the step of determining if said data item needs to be updated; and managing a plugin associated with said data item to update said data item.
20. The computer based method of claim 16 , further comprising loading and launching said plugin associated with said data item; and retrieving said unstructured data in response to said data item to be monitored.
21. The computer based method of claim 16 , further comprising filtering said unstructured data in response to a plurality of data item parameters.
22. The computer based method of claims 21 , wherein the filtering step further comprises the step of filtering said unstructured data in response to one or more data item parameters received from said plurality of users.
23. The computer based method of claim 16 , further comprising the step of filtering said structured data in response to a plurality of data item parameters.
24. The computer based method of claim 16 , further comprising the step of displaying said stored structured data received in response to said data item associated with a user.
25. The computer based method of claim 24 , further comprising the step of displaying said stored structured data in a plurality of graphical representations selectable by said user.
26. The computer based method claim 24 , wherein the step of displaying said stored structure further comprises the step of displaying said structured data in one or more of the following graphical representations: displaying said structured data related to one data item associated with said user, displaying structured data related to two or more data items associated with said user; displaying structured data associated with said user from a single data source; displaying structured data associated said user from multiple data sources.
27. The computer based method of claim 16 , further comprising the step of generating a watch group by a user, said watch group comprising one or more data items relating to the subject matter; said associated data sources; and said graphical representations of said structured data.
28. The computer based method of claim 27 , wherein the step of generating a watch group further comprises enabling said user to share said watch group with another user.