System and method for providing a topic-directed search
View Patent ↗A system and method for providing a topic-directed search is provided, which advantageously harnesses user-provided topical indexes and an ability to characterize indexes according to how articles fall under their topical organizations. A corpus of articles and an index that includes topics from the articles is maintained. For each topic, a coarse-grained topic model is built, which includes the characteristic words included in the articles relating to the topic and scores assigned to the characteristic words. A search query is executed against the index. The topics that match the search terms are chosen by their scores. The topics that match the coarse-grained topic models and the articles corresponding to the search query are presented. In contrast to conventional search engines, search results are organized according to topic and search results can be offered across multiple indexes, where part of returned results are selected from most-relevant indexes with their most-relevant topics.
1. A computer-implemented system for providing a topic-directed search, comprising:
a plurality of evergreen indexes for topically-limited subject areas, each of the subject areas comprising articles of digital information and for each of the evergreen indexes, comprising:
a hierarchy of topics; and
a fine-grained topic model paired to each of the topics in the topic hierarchy, each of the fine-grained topic models comprising a pattern evaluable against the articles, wherein the pattern identifies such articles matching the fine-grained topic model's topic, that are both electronically-stored in persistent storage;
a computer comprising a processor and memory within which code for execution by the processor is stored, comprising:
a topic builder that, for each topic in each of the evergreen indexes, maintains a coarse-grained topic model comprising:
a characteristic word extractor that extracts characteristic words from the articles identified by the fine-grained topic models of each topic; and
a characteristic word scorer that assigns scores to the characteristic words; and
a search engine that specifies a topic search comprising search terms, and that evaluates the coarse-grained topic model of each of the topics in the evergreen indexes by matching the search terms and accumulating their scores;
a user interface that visually presents the evergreen indexes comprised of the topics that comprise the coarse-grained topic models having the best scores; and further comprising at least one of:
a display module comprising code for execution by the processor that comprises:
a topic emphasizer highlighting the topics in the best scoring evergreen indexes as presented in the user interface, comprising at least one of emphasizing a to percentile of the articles corresponding to the search query, emphasizing the topics whose matching coarse-grained topic models have a cumulative score exceeding a threshold, and emphasizing the topics comprising top-scoring characteristic words that match the search terms; and
a topic elider eliding the topics presented in the user interface that fall below a set matching score;
an index identifier identifying the best scoring evergreen indexes by one or more of name, theme, and augmented community of online users; and
a pattern evaluator evaluating each of the patterns for the fine-grained topic models comprised in the best scoring evergreen indexes against the digital information, the fine-grained topic models corresponding to the topics that comprise the coarse-grained topic models having the best scores, and providing access to at least a part of only that digital information that matches at least one of the patterns.
2. A system according to claim 1 , further comprising:
a model builder comprising code for execution by the processor that comprises:
a random sampler selecting a random sampling of the articles matching a topic; and
a scoring module determining frequencies of occurrence of the characteristic words comprised in the articles in the random sampling and in positive training examples, and identifying a ratio of the frequencies of occurrence for the characteristic words comprised in the random sampling and the positive training examples,
wherein the ratios of the characteristic words are included as the scores of the coarse-grained topic models.
3. A system according to claim 2 , further comprising:
a pre-computing monitor comprising code for execution by the processor that comprises:
a counter monitoring a number of articles comprised in the topics of the index; and
an analyzer periodically re-determining the frequencies of occurrence of the characteristic words comprised in the articles in the random sampling when the number of articles has changed by a predetermined amount.
4. A system according to claim 2 , wherein a sampling of articles matching the fine-grained topic models for each topic is selected in lieu of the positive training examples.
5. A system according to claim 1 , further comprising:
a scoring module comprising code for execution by the processor that comprises:
an evaluator finding the highest score for the coarse-grained topic models; and
a normalizer normalizing the scores of the remaining coarse-grained topic models against the highest score.
6. A system according to claim 1 , further comprising:
a scoring module comprising code for execution by the processor that comprises:
a comparison module, for each topic, comparing the score of each coarse-grained topic model corresponding to one of the search terms to a threshold; and
an evaluator either discounting or discarding the topic when at least one of the scores fails to exceed the threshold.
7. A system according to claim 1 , further comprising:
a scoring module comprising code for execution by the processor that comprises:
a score adjuster adjusting the scores of the coarse-grained topic models, comprising one or more of:
an infrequent word evaluator reducing the score for each characteristic word appearing fewer than a minimum number of times in the articles;
a short word evaluator reducing the score for each characteristic word comprising a length of less than a minimum number of characters;
a label word evaluator increasing the score for each characteristic word appearing in an index label of one or more of the articles; and
a neighboring word evaluator increasing the score of each characteristic word either neighboring or appearing adjacent to another characteristic word appearing in an index label of one or more of the articles.
8. A computer-implemented method for providing a topic-directed search, comprising:
maintaining a plurality of evergreen indexes for topically-limited subject areas, each of the subject areas comprising articles of digital information and for each of the evergreen indexes, comprising:
specifying a hierarchy of topics; and
pairing a fine-grained topic model to each of the topics in the topic hierarchy, each of the fine-grained topic models comprising a pattern evaluable against the articles, wherein the pattern identifies such articles matching the fine-grained topic model's topic;
for each topic in each of the evergreen indexes, maintaining a coarse-grained topic model comprising:
specifying characteristic words extracted from the articles identified by the fine-grained topic models of each topic; and
assigning scores to the characteristic words;
specifying a topic search comprising search terms;
evaluating the coarse-grained topic model of each of the topics in the evergreen indexes by matching the search terms and accumulating their scores;
presenting the evergreen indexes comprised of the topics that comprise the coarse-grained topic models having the best scores through a user interface; and further comprising at least one of:
highlighting the topics in the best scoring evergreen indexes as presented in the user interface, comprising at least one of:
emphasizing a to percentile of the articles corresponding to the search query;
emphasizing the topics whose matching coarse-grained topic models have a cumulative score exceeding a threshold; and
emphasizing the topics comprising top-scoring characteristic words that match the search terms;
eliding the topics presented in the user interface that fall below a set matching score;
identifying the best scoring evergreen indexes by one or more of name, theme, and augmented community of online users; and
evaluating each of the patterns for the fine-grained topic models comprised in the best scoring evergreen indexes against the digital information, the fine-grained topic models corresponding to the topics that comprise the coarse-grained topic models having the best scores, and providing access to at least a part of only that digital information that matches at least one of the patterns.
9. A method according to claim 8 , further comprising:
selecting a random sampling of the articles matching a topic;
determining frequencies of occurrence of the characteristic words comprised in the articles in the random sampling and in positive training examples;
identifying a ratio of the frequencies of occurrence for the characteristic words comprised in the random sampling and the positive training examples; and
including the ratios of the characteristic words as the scores of the coarse-grained topic models.
10. A method according to claim 9 , further comprising:
monitoring a number of articles comprised in the topics of the index; and
periodically re-determining the frequencies of occurrence of the characteristic words comprised in the articles in the random sampling when the number of articles has changed by a predetermined amount.
11. A method according to claim 9 , further comprising:
selecting a sampling of articles matching the fine-grained topic models for each topic in lieu of the positive training examples.
12. A method according to claim 8 , further comprising:
finding the highest score for the coarse-grained topic models; and
normalizing the scores of the remaining coarse-grained topic models against the highest score.
13. A method according to claim 8 , further comprising:
for each topic, comparing the score of each coarse-grained topic model corresponding to one of the search terms to a threshold; and
either discounting or discarding the topic when at least one of the scores fails to exceed the threshold.
14. A method according to claim 8 , further comprising:
adjusting the scores of the coarse-grained topic models, comprising one or more of:
reducing the score for each characteristic word appearing fewer than a minimum number of times in the articles;
reducing the score for each characteristic word comprising a length of less than a minimum number of characters;
increasing the score for each characteristic word appearing in an index label of one or more of the articles; and
increasing the score of each characteristic word either neighboring or appearing adjacent to another characteristic word appearing in an index label of one or more of the articles.
15. A system according to claim 1 , further comprising:
a pattern evaluator evaluating each of the patterns for the fine-grained topic models comprised in the best scoring evergreen indexes against the articles of digital information, the evaluated fine-grained topic models corresponding to the topics associated with the coarse-grained topic models having the best scores, and providing access to at least a part of the set of articles in the digital information that match at least one of the patterns.
16. A method according to claim 8 , further comprising:
evaluating each of the patterns for the fine-grained topic models comprised in the best scoring evergreen indexes against the articles of digital information, the evaluated fine-grained topic models corresponding to the topics associated with the coarse-grained topic models having the best scores; and
providing access to at least a part of the set of articles in the digital information that match at least one of the patterns.