LARGE SCALE ANALYTICAL REPORTING FROM WEB CONTENT
An analysis system is configured to perform quantitative (e.g., statistical) and/or qualitative analysis of large numbers of documents, files, etc., such as web pages, images on web pages, social medial pages, or other documents available via the Internet, an extranet, an Intranet, etc. In some embodiments, the large numbers of documents, files, etc., may be treated as a data set that can be quantitatively analyzed and reports of such analyses may be made electronically available to a user.
1 . A method for quantitatively and/or qualitatively analyzing a plurality of documents available on a communication network, comprising:
generating, at one or more computing devices, inferred metadata regarding documents in the plurality of documents available on the communication network at least by (i) using a search engine index that indexes the plurality documents, and (ii) analyzing one or more of (a) contents of documents in the plurality of documents, (b) metadata included in documents in the plurality of documents, (c) data associated with documents in the plurality of documents and external to the documents, (d) previously generated inferred metadata regarding documents in the plurality of documents;
responsive to a request to perform a quantitative and/or qualitative analysis of the plurality of documents, performing, at one or more computing devices, the requested analysis using (i) the search engine index, and (ii) the inferred metadata; and
generating, at one or more computing devices, a report that includes results of the requested analysis.
2 . A method according to claim 1 , wherein at least some of the inferred metadata is generated responsive to the request to perform the analysis.
3 . A method according to claim 1 , wherein at least some of the inferred metadata is generated prior to receiving the request to perform the analysis.
4 . A method according to claim 1 , wherein generating the inferred metadata comprises determining for each document in at least a subset of the documents at least one of (i) a topic of the document, and (ii) and a category of the document.
5 . A method according to claim 4 , wherein determining a topic of a document comprises at least one of (i) analyzing a title in the document, (ii) analyzing headings in the document, (iii) analyzing first sentences in paragraphs of the document, (iv) analyzing the frequency in which words are utilized in the document, and (v) identifying words that correspond to an entity known to an analysis system.
6 . A method according to claim 4 , wherein determining a category of a document comprises determining that the document is at least one of (i) a page of a personal web site, (ii) a page of a professional web site, (iii) a blog, (iv) concerning a product or service, and (v) a contacts page.
7 . A method according to claim 1 , wherein performing the requested analysis comprises:
identifying documents, in the plurality of documents, relevant to the analysis using (i) the search engine index, and (ii) the inferred metadata; and
performing the requested quantitative and/or qualitative analysis on the identified relevant documents.
8 . A method according to claim 7 , wherein performing the requested quantitative and/or qualitative analysis on the identified relevant documents comprises analyzing contents of the identified relevant documents.
9 . A method according to claim 7 , wherein performing the requested quantitative and/or qualitative analysis on the identified relevant documents comprises:
determining a subset of the identified documents that satisfy a criterion; and
determining a quantity of documents in the subset.
10 . A method according to claim 1 , wherein performing the requested analysis comprises:
performing the requested analysis using results of a previously performed analysis.
11 . A method according to claim 1 , further comprising generating the search engine index.
12 . A method according to claim 1 , further comprising adding inferred metadata to the search engine index.
13 . A method according to claim 1 , wherein:
the communication network includes the Internet; and
the search engine index is an Internet search engine index; and
analyzing contents of documents in the plurality of documents comprises at least one of (i) obtaining at least a first subset of the documents via the Internet, and (ii) obtaining at least a second subset of the documents via a cache generated by the Internet search engine.
14 . A system for quantitatively and/or qualitatively analyzing a plurality of documents available on a communication network, the system comprising:
one or more computing devices configured to:
generate inferred metadata regarding documents in the plurality of documents available on the communication network at least by (i) using a search engine index that indexes the plurality documents, and (ii) analyzing one or more of (a) contents of documents in the plurality of documents, (b) metadata included in documents in the plurality of documents, (c) data associated with documents in the plurality of documents and external to the documents, (d) previously generated inferred metadata regarding documents in the plurality of documents,
perform, responsive to a request to perform a quantitative and/or qualitative analysis of the plurality of documents, the requested analysis using (i) the search engine index, and (ii) the inferred metadata, and
generate a report that includes results of the requested analysis.
15 . A system according to claim 14 , wherein the one or more computing devices are configured to generate at least some of the inferred metadata responsive to the request to perform the analysis.
16 . A system according to claim 14 , wherein the one or more computing devices are configured to generate at least some of the inferred metadata prior to receiving the request to perform the analysis.
17 . A system according to claim 14 , wherein the one or more computing devices are configured to generate inferred metadata that includes, for each document in at least a subset of the documents, at least one of (i) a topic of the document, and (ii) and a category of the document.
18 . A system according to claim 17 , wherein the one or more computing devices are configured to determine a topic of a document at least by one or more of (i) analyzing a title in the document, (ii) analyzing headings in the document, (iii) analyzing first sentences in paragraphs of the document, (iv) analyzing the frequency in which words are utilized in the document, and (v) identifying words that correspond to an entity known to an analysis system.
19 . A system according to claim 17 , wherein the one or more computing devices are configured to determine a category of a document at least by determining that the document is at least one of (i) a page of a personal web site, (ii) a page of a professional web site, (iii) a blog, (iv) concerning a product or service, and (v) a contacts page.
20 . A system according to claim 14 , wherein the one or more computing devices are configured to perform the requested analysis at least by:
identifying documents, in the plurality of documents, relevant to the analysis using (i) the search engine index, and (ii) the inferred metadata; and
performing the requested quantitative and/or qualitative analysis on the identified relevant documents.
21 . A system according to claim 20 , wherein the one or more computing devices are configured to perform the requested quantitative and/or qualitative analysis on the identified relevant documents at least by analyzing contents of the identified relevant documents.
22 . A system according to claim 20 , wherein the one or more computing devices are configured to perform the requested quantitative and/or qualitative analysis on the identified relevant documents at least by:
determining a subset of the identified documents that satisfy a criterion; and
determining a quantity of documents in the subset.
23 . A system according to claim 14 , wherein the one or more computing devices are configured to perform the requested analysis at least by:
performing the requested analysis using results of a previously performed analysis.
24 . A system according to claim 14 , wherein the one or more computing devices are configured to generate the search engine index.
25 . A system according to claim 14 , wherein the one or more computing devices are configured to add inferred metadata to the search engine index.
26 . A system according to claim 14 , wherein:
the communication network includes the Internet; and
the search engine index is an Internet search engine index; and
the one or more computing devices are configured to analyze contents of documents in the plurality of documents at least by one or both of (i) obtaining at least a first subset of the documents via the Internet, and (ii) obtaining at least a second subset of the documents via a cache generated by the Internet search engine.
27 . A system according to claim 14 , further comprising a database to store the inferred metadata.
28 . A system according to claim 27 , wherein the database also stores the search engine index.
29 . A tangible, non-transitory, computer readable medium that stores machine readable instructions that, when executed by one or more computing devices, cause the one or more computing devices to:
generate inferred metadata regarding documents in the plurality of documents available on the communication network at least by (i) using a search engine index that indexes the plurality documents, and (ii) analyzing one or more of (a) contents of documents in the plurality of documents, (b) metadata included in documents in the plurality of documents, (c) data associated with documents in the plurality of documents and external to the documents, (d) previously generated inferred metadata regarding documents in the plurality of documents;
responsive to a request to perform a quantitative and/or qualitative analysis of the plurality of documents, perform the requested analysis using (i) the search engine index, and (ii) the inferred metadata; and
generate a report that includes results of the requested analysis.