System and method for collecting and processing data
A system and method for collecting and processing data over a communications network. A data mining marshaller module associates each plugin to a particular data source and manages the plugin to periodically retrieve unstructured data from the data source based on a plurality of data items to be monitored on behalf of a plurality of users. The plugins convert unstructured data received from the data sources to structured data and the data marshaller module stores the structured data in a database. This enables the system and method to aggregate and display the structured data in multiple graphical representations according to the user's preference.
1. A system for collecting and processing data over a communications network, comprising:
a processor based web server comprising:
a plurality of plugins stored in the server, each configured to retrieve unstructured data from a data source on behalf of a plurality of users over the communications network and to convert the unstructured data by parsing the unstructured data into structured data comprising historical data, updated data and analytical graphical representation of said data;
an activation module for receiving each user's selection of one or more data items, each data item relating to a subject matter to be monitored from a client device associated with said each user over said communications network; and
a data mining marshaller (DMM) module for associating a plugin to each data item selected by said plurality of users, associating the plugin to an associated data source, associating the plugin to each user who have selected a similar data item to be monitored, and managing each plugin to retrieve the unstructured data from the associated data source to enable multiple users to share the structured data based on shared interest in similar subject matter; and
a database to store the structured data received from said plurality of plugins.
2. The system of claim 1 , wherein the DMM module determines if a data item needs to be updated; and to manage a plugin associated with the data item to update the data item.
3. The system of claim 1 , wherein the DMM module is configured to rank TV shows based on engagement correlated with expressions occurring in social media sources.
4. The system of claim 1 , wherein the DMM module is configured to rank TV shows based on engagement correlated with blog comments.
5. The system of claim 1 , wherein the DMM module is configured to measure trends and impact of TV shows in social media.
6. The system of claim 1 , wherein the processor based web server further comprises a display navigation module to display the stored structured data received in response to a data item associated with a user in a plurality of graphical representations selectable by the user.
7. The system of claim 6 , wherein the navigation module is configured to drill down to the associated data sources from data points on a graph selected by the user, or overlay data from different data sources selected by the user on one graph or on side-by-side graphs.
8. A non-transitory computer-readable medium comprising a code for collecting and processing data over a communications network, the code comprising instructions for:
configuring a plurality of plugins stored in processor based server, each plugin configured to retrieve unstructured data from a data source over the communications network and to convert the unstructured data by parsing the unstructured data into structured data comprising historical data, updated data and analytical graphical representation of said data;
receiving each user's selection of one or more data items relating to a subject matter to be monitored from a client device associated with said each user over the communications network;
associating a plugin to each data item selected by said plurality of users;
associating the plugin to an associated data source;
associating the plugin to each user who have selected a similar data item to be monitored;
managing each plugin to retrieve the unstructured data from the associated data source to enable multiple users to share the structured data based on shared interest in similar subject matter; and
storing said structured data received from said plurality of plugins in a database.
9. The computer-readable medium of claim 8 , wherein the code further comprises instructions for determining if a data item needs to be updated; and managing a plugin associated with the data item to update the data item.
10. The computer-readable medium of claim 8 , wherein the code further comprises instructions for ranking TV shows based on engagement correlated with expressions occurring in social media sources or for ranking TV shows based on engagement correlated with blog comments.
11. The computer-readable medium of claim 8 , wherein the code further comprises instructions for measuring trends and impact of TV shows in social media.
12. The computer-readable medium of claim 8 , wherein the code further comprises instructions for displaying the stored structured data received in response to a data item associated with a user in a plurality of graphical representations selectable by the user.
13. The computer-readable medium of claim 12 , wherein the code further comprises instructions for drilling down to the associated data sources from data points on a graph selected by the user, or for overlaying data from different data sources selected by the user on one graph or on side-by-side graphs.
14. A computer based method for collecting and processing data over a communications network, comprising the steps of:
configuring a plurality of plugins stored in processor based server, each plugin configured to retrieve unstructured data from a data source over the communications network and to convert the unstructured data by parsing the unstructured data into structured data comprising historical data, updated data and analytical graphical representation of said data;
receiving each user's selection of one or more data items relating to a subject matter to be monitored from a client device associated with said each user over the communications network;
associating a plugin to each data item selected by said plurality of users;
associating the plugin to an associated data source;
associating the plugin to each user who have selected a similar data item to be monitored;
managing each plugin to retrieve the unstructured data from the associated data source to enable multiple users to share the structured data based on shared interest in similar subject matter; and
storing said structured data received from said plurality of plugins in a database.
15. The method of claim 14 , further comprising the step of determining if a data item needs to be updated; and managing a plugin associated with the data item to update the data item.
16. The method of claim 14 , further comprising the step of ranking TV shows based on engagement correlated with expressions occurring in social media sources.
17. The method of claim 14 , further comprising the step of ranking TV shows based on engagement correlated with blog comments.
18. The method of claim 14 , further comprising the step of measuring trends and impact of TV shows in social media.
19. The method of claim 14 , further comprising the step of displaying the stored structured data received in response to a data item associated with a user in a plurality of graphical representations selectable by the user.
20. The method of claim 19 , further comprising the step of drilling down to the associated data sources from data points on a graph selected by the user, or for overlaying data from different data sources selected by the user on one graph or on side-by-side graphs.