Method and subsystem for information acquisition and aggregation to facilitate ontology and language-model generation within a content-search-service system
View Patent ↗Various embodiments of the present invention include information-aggregation-and-classification components of content-search-service systems which acquire information from information sources, aggregate and normalize the acquired information, and classify the acquired information prior to storing the normalized and classified information for use by language-model-builder components and ontology-builder components of the content-search-service systems. Additional embodiments of the present invention include the ontology-builder components, which builds ontologies from the normalized and classified information for specific dates, date/times, date ranges, or date/time ranges and for specific categories.
1. A computer-based information-aggregation-and-classification system comprising:
a document-classifier;
a document-classifier-constructor subcomponent that uses a collection of categorized and processed documents to construct the document-classifier; and
an information-aggregator subcomponent that:
obtains uncategorized documents from electronic information sources;
filters the uncategorized documents to produce a set of normalized text documents by, for each uncategorized document:
removing non-text items and symbols from the uncategorized document to produce a text document having text;
parsing the text to organize the text of the text document into paragraphs and sentences within paragraphs to produce a paragraph-and-sentence-delimited text document;
applying language-parsing routines to the paragraph-and-sentence-delimited text document to remove non-noun terms from the text to produce a noun-and-noun-phrase-culled text document; and
filtering non-useful terms from normalized terms in the noun-and-noun-phrase-only text document to produce a normalized text document;
applies an indexer to the set of normalized text documents to generate a term vector for each normalized text document in the set of normalized text documents;
inputs the term vector computed for each normalized text document in the set of normalized text documents to assign a category to the normalized text document; and
aggregates normalized text documents of each different category together to produce a normalized-text-document package for each category.
2. The computer-based information-aggregation-and-classification system of claim 1 , wherein the document-classifier is a naive Bayesian document classifier.
3. The computer-based information-aggregation-and-classification system of claim 1 , wherein the document-classifier-constructor subcomponent:
obtains documents, to each of which a category is assigned, from electronic information sources;
filters the documents to produce a second set of normalized text documents;
applies an indexer to the second set of normalized text documents to generate a term vector for each normalized text document in the second set of normalized text documents;
computes cumulative term distributions for documents of each category; and
inputs each normalized text document, together with a term vector computed for the normalized text document and the category assigned to the text document, and the cumulative term distributions for documents of each category to the document classifier to train the document classifier to recognize a category to assign to an input normalized-text-document/term-vector pair.
4. The computer-based information-aggregation-and-classification system of claim 1 , wherein the information-aggregator subcomponent uses a web crawler to obtain uncategorized documents from electronic information sources.
5. The computer-based information-aggregation-and-classification system of claim 1 , wherein each term vector contains an occurrence count for each term in the normalized text document for which the term vector was computed.
6. The computer-based information-aggregation-and-classification system of claim 1 , wherein each normalized-text-document package is date-stamped or date/time-stamped to indicate the date or date and time when the normalized-text-document package was prepared.
7. A method for preparing normalized-text-document packages, the method comprising:
using a collection of categorized and processed documents to construct a document-classifier;
obtaining uncategorized documents from electronic information sources;
filtering the uncategorized documents to produce a set of normalized text documents by, for each uncategorized document:
removing non-text items and symbols from the uncategorized document to produce a text document having text;
parsing the text to organize the text of the text document into paragraphs and sentences within paragraphs to produce a paragraph-and-sentence-delimited text document;
applying language-parsing routines to the paragraph-and-sentence-delimited text document to remove non-noun terms from the text to produce a noun-and-noun-phrase-culled text document; and
filtering non-useful terms from normalized terms in the noun-and-noun-phrase-only text document to produce a normalized text document;
applying an indexer to the set of normalized text documents to generate a term vector for each normalized text document in the set of normalized text documents;
inputting the term vector computed for each normalized text document in the set of normalized text documents to assign a category to the normalized text document; and
aggregating normalized text documents of each different category together to produce a normalized-text-document package for each category.
8. The method of claim 7 , wherein the document-classifier is a naive Bayesian document classifier.
9. The method of claim 7 , wherein constructing the document-classifier further includes:
obtaining documents, to each of which a category is assigned, from electronic information sources;
filtering the documents to produce a second set of normalized text documents;
applying an indexer to the second set of normalized text documents to generate a term vector for each normalized text document in the second set of normalized text documents;
computing cumulative term distributions for documents of each category; and
inputting each normalized text document, together with a term vector computed for the normalized text document and the category assigned to the text document, and the cumulative term distributions for documents of each category to the document classifier to train the document classifier to recognize a category to assign to an input normalized-text-document/term-vector pair.
10. The method of claim 7 , wherein obtaining uncategorized documents from electronic information sources further comprises using a web crawler.
11. The method of claim 7 , wherein each term vector contains an occurrence count for each term in the normalized text document for which the term vector was computed.
12. The method of claim 7 , wherein each normalized-text-document package is date-stamped or date/time-stamped to indicate the date or date and time when the normalized-text-document package was prepared.
13. A machine-readable medium having instructions stored thereon which, when executed by a processor, cause one or more machines to:
use a collection of categorized and processed documents to construct a document-classifier;
obtain uncategorized documents from electronic information sources;
filter the uncategorized documents to produce a set of normalized text documents by, for each uncategorized document:
removing non-text items and symbols from the uncategorized document to produce a text document having text;
parsing the text to organize the text of the text document into paragraphs and sentences within paragraphs to produce a paragraph-and-sentence-delimited text document;
applying language-parsing routines to the paragraph-and-sentence-delimited text document to remove non-noun terms from the text to produce a noun-and-noun-phrase-culled text document; and
filtering non-useful terms from normalized terms in the noun-and-noun-phrase-only text document to produce a normalized text document;
apply an indexer to the set of normalized text documents to generate a term vector for each normalized text document in the set of normalized text documents;
input the term vector computed for each normalized text document in the set of normalized text documents to assign a category to the normalized text document; and
aggregate normalized text documents of each different category together to produce a normalized-text-document package for each category.
14. The machine-readable medium of claim 13 , wherein the document-classifier is a naive Bayesian document classifier.
15. The machine-readable medium of claim 13 , wherein constructing the document-classifier further includes:
obtaining documents, to each of which a category is assigned, from electronic information sources;
filtering the documents to produce a second set of normalized text documents;
applying an indexer to the second set of normalized text documents to generate a term vector for each normalized text document in the second set of normalized text documents;
computing cumulative term distributions for documents of each category; and
inputting each normalized text document, together with a term vector computed for the normalized text document and the category assigned to the text document, and the cumulative term distributions for documents of each category to the document classifier to train the document classifier to recognize a category to assign to an input normalized-text-document/term-vector pair.
16. The machine-readable medium of claim 13 , wherein obtaining uncategorized documents from electronic information sources further comprises using a web crawler.
17. The machine-readable medium of claim 13 , wherein each term vector contains an occurrence count for each term in the normalized text document for which the term vector was computed.