Automatic generation of stop word lists for information retrieval and analysis
Methods and systems for automatically generating lists of stop words for information retrieval and analysis. Generation of the stop words can include providing a corpus of documents and a plurality of keywords. From the corpus of documents, a term list of all terms is constructed and both a keyword adjacency frequency and a keyword frequency are determined. If a ratio of the keyword adjacency frequency to the keyword frequency for a particular term on the term list is less than a predetermined value, then that term is excluded from the term list. The resulting term list is truncated based on predetermined criteria to form a stop word list.
1. A computer-implemented method of generating a stop word list for information retrieval and analysis, the method comprising:
providing a corpus of documents and a plurality of keywords;
constructing a term list of all terms in the corpus of documents;
determining keyword adjacency frequency of each term in the corpus of documents, wherein the keyword adjacency frequency comprises the number of times a term occurs adjacent to one of the keywords;
determining keyword frequency of each term in the corpus of documents,
wherein the keyword frequency comprises the number of times a term occurs within one of the keywords;
excluding from the term list each term having a ratio of keyword adjacency frequency to keyword frequency that is less than a predetermined value; and
truncating the term list based on a predetermined criteria to form the stop word list.
2. The computer-implemented method of claim 1 , wherein keywords are provided for each document in the corpus of documents.
3. The computer-implemented method of claim 1 , wherein the predetermined value of the ratio of the keyword adjacency frequency to keyword frequency is 1.
4. The computer-implemented method of claim 1 , wherein the predetermined criteria comprise a document frequency value that is less than a threshold value.
5. The computer-implemented method of claim 1 , wherein the predetermined criteria comprise a predetermined fixed length for the stop word list.
6. The computer-implemented method of claim 1 , further comprising applying the stop word list to parse a second corpus of documents.
7. A system for generating a stop word list for information retrieval and analysis, the system comprising;
a storage device, a communications interface, or both providing a corpus of documents and a plurality of keywords; and
a processor programmed to:
construct a term list of all terms in the corpus of documents;
determine keyword adjacency frequency of each term in the corpus of documents, wherein the keyword adjacency frequency comprises the number of times a term occurs adjacent to one of the keywords;
determine keyword frequency of each term in the corpus of documents, wherein the keyword frequency comprises the number of times a term occurs within one of the keywords;
exclude terms from the term list if the ratio of keyword adjacency frequency to keyword frequency is less than a predetermined value; and
truncate the term list based on predetermined criteria to form the stop word list.
8. The system of claim 7 , wherein keywords are provided for each document in the corpus of documents.
9. The system of claim 7 , wherein the predetermined value of the ratio of the keyword adjacency frequency to keyword frequency is 1.
10. The system of claim 7 , wherein the predetermined criteria comprises a document frequency value that is less than a threshold value.
11. The system of claim 7 , wherein the predetermined criteria comprises a predetermined fixed length for the stop word list.
12. The system of claim 7 , wherein the processor is further configured to apply the stop word list to parse a second corpus of documents.