System and method of ad-hoc analysis of data
View Patent ↗It is a first aspect of the present invention to provide a computer implemented method of performing ad-hoc analysis including the steps of: generating a text index of the textual information items, generating a metadata lookup structure based, at least in part, on the text index, searching the text index using a search query, compiling results of the text index search into aggregate information related to characteristics of the search results from the metadata items associated with the textual information items in the search results from the metadata lookup structure, and reporting the aggregate information.
1. A computer implemented method for performing ad-hoc analysis, the method comprising:
generating, using a processor, a text index of a plurality of text items found in at least one electronic information source;
generating, using a processor, a metadata lookup structure based, at least in part, on the text index, the metadata lookup structure including one or more metadata items associated with each of the text items, wherein generating the metadata lookup structure comprises interrogating the text index for one or more text items matching the one or more metadata items;
facilitating a user search of the text index using one or more search queries, the searching producing search results including one or more text items matching the one or more search queries;
retrieving one or more metadata items using the metadata lookup structure based on the search results;
compiling results of the text index search into aggregated information based on the retrieved metadata items associated with each of the one or more text items in the search results, wherein the metadata lookup structure is used to retrieve the one or more metadata items associated with the one or more text items found in the search results; and
reporting the aggregated information.
2. The method of claim 1 , further comprising:
prior to generating the text index, accessing a plurality of information sources for a plurality of text items.
3. The method of claim 1 , wherein generating a metadata lookup structure based, at least in part, on the text index includes:
generating a plurality of metadata identifiers (IDs), each metadata ID associated with at least a type of metadata;
analyzing each text item to determine which metadata ID(s) are associated with the respective text item; and
mapping each text item with the respective metadata ID(s) determined for it in the analyzing.
4. The method of claim 1 , wherein generating a metadata lookup structure based, at least in part, on the text index includes:
generating one or more metadata items associated with the text items;
determining a quantity of the one or more metadata items; and
dynamically allocating a portion of a computer memory component based, at least in part, on the determined quantity.
5. The method of claim 1 , wherein generating a metadata lookup structure based, at least in part, on the text index includes:
generating one or more metadata items associated with the text items, wherein the number of metadata items generated is not the same for all of the text items;
determining a quantity of the one or more metadata items; and
dynamically allocating a portion of a computer memory component based, at least in part, on the determined quantity.
6. The method of claim 1 , wherein the one or more metadata items includes at least one of date information, link information, author information, keyword information, sentiment information, demographic information, entity information, and language information.
7. The method of claim 6 , wherein the link information includes a Uniform Resource Locator.
8. The method of claim 6 , wherein the demographic information is generated and includes at least one of age and gender of a text item author.
9. The method of claim 6 , wherein the language information includes one or more language specific annotation.
10. The method of claim 9 , wherein the language specific annotation is provided by the text items.
11. The method of claim 9 , wherein the language specific annotation is determined by analyzing the text items.
12. The method of claim 1 , wherein the text items include electronic data from one or more of Internet message boards, blogs and news groups.
13. The method of claim 1 , wherein the aggregate information includes at least one of date information, link information, author information, keyword information, sentiment information, demographic information, entity information, and language information pertaining to the search results.
14. The method of claim 1 , wherein the text index is updated after a time period.
15. The method of claim 14 , wherein the time period is between five and fifteen minutes.
16. A system for performing ad-hoc analysis, comprising:
a computer server having access to one or more information sources, the one or more information sources including a plurality of text items; and
a user computer device linked via one or more data links to the computer server, the user computer device including software configured to perform:
generating a text index of the text items;
generating a metadata lookup structure based, at least in part, on the text index, the metadata lookup structure including one or more metadata items associated with each of the text items;
facilitating a search of the text index using one or more search queries, the searching producing search results including one or more text items matching at least one of the one or more search queries;
retrieving one or more metadata items using the metadata lookup structure based on the search results;
compiling results of the text index search into aggregated information based on the retrieved metadata items associated with each of the one or more text items in the search results; and
reporting the aggregated information.
17. The system of claim 16 , wherein generating a metadata lookup structure based, at least in part, on the text index includes:
generating a plurality of metadata IDs, each metadata ID associated with at least a type of metadata;
analyzing each text item to determine which metadata ID(s) are associated with the respective text item; and
mapping each text item with the respective metadata ID(s) determined for it in the analyzing.
18. The system of claim 16 , wherein generating a metadata lookup structure based, at least in part, on the text index includes:
generating one or more metadata items associated with the text items;
determining a quantity of the one or more metadata items; and
dynamically allocating a portion of a computer memory component based, at least in part, on the determined quantity.
19. The system of claim 16 , wherein generating a metadata lookup structure based, at least in part, on the text index includes:
generating one or more metadata items associated with the text items, wherein the number of metadata items generated is not the same for all of the text items;
determining a quantity of the one or more metadata items; and
dynamically allocating a portion of a computer memory component based, at least in part, on the determined quantity.
20. The system of claim 16 , wherein the one or more metadata items includes at least one of date information, link information, author information, keyword information, sentiment information, demographic information, entity information, and language information.
21. The system of claim 16 , wherein the aggregate information includes at least one of date information, link information, author information, keyword information, sentiment information, demographic information, entity information, and language information pertaining to the search results.
22. The system of claim 16 , wherein the text index is updated after a time period.
23. The system of claim 22 , wherein the time period is between five and fifteen minutes.