Semi-automatic index term augmentation in document retrieval
Disclosed are methods and systems for indexing or retrieving materials accessible through computer networks.
1. A method implemented by one or more computing devices, wherein the one or more computing devices are configured to perform the following:
selecting a first document;
generating a query containing at least one term from the selected first document;
applying the generated query to a plurality of documents to define a subset of the plurality of documents, wherein the defined subset of the plurality of documents constitutes those documents of the plurality of documents that contain the at least one term and meet a predetermined threshold of query relevance;
determining additional terms based on the defined subset of the plurality of documents, including:
determining a co-occurrence metric of the at least one term from the selected first document with each term of the defined subset of the plurality of documents;
determining a frequency score using the determined co-occurrence metric for each term of the defined subset of the plurality of documents; and
selecting a subset of terms of the defined subset of the plurality of documents based on the determined frequency score for each term;
assigning the selected subset of terms to the selected first document; and
storing the selected first document in a storage system that is remotely accessible via a network.
2. The method of claim 1 , wherein the at least one term includes all terms in the selected first document.
3. The method of claim 1 , wherein query relevance is determined using at least one of term frequency scoring and inverted document frequency scoring, and the predetermined threshold of query relevance is a maximum number of highest scoring documents.
4. The method of claim 3 , wherein the query relevance is determined using Robertson's term frequency scoring.
5. The method of claim 1 , wherein the plurality of documents are Web pages.
6. The method of claim 1 , wherein the plurality of documents are Web sites.
7. The method of claim 1 , wherein terms do not include stop words.
8. The method of claim 1 , wherein selecting the subset of terms based on the determined frequency score for each term includes selecting terms having the determined frequency score above a predetermined term assignment threshold.
9. The method of claim 8 , wherein the predetermined term assignment threshold is a single highest scoring term.
10. The method of claim 1 , wherein assigning the selected subset of terms includes inserting the selected subset of terms into a portion of the selected first document.
11. The method of claim 10 , wherein the portion of the selected first document is a segment of the selected first document for storing terms to improve document retrieval methods.
12. The method of claim 1 , wherein the plurality of documents has previously had index terms assigned to the plurality of documents by another method.
13. The method of claim 12 , wherein weighting is applied to the index terms assigned to the plurality of documents when applying the generated query to the plurality of documents.
14. The method of claim 1 , wherein the network includes the Internet.