Method for identifying related pages in a hyperlinked database
A method is described for identifying related pages among a plurality of pages in a linked database such as the World Wide Web. An initial page is selected from the plurality of pages. Pages linked to the initial page are represented as a graph in a memory. The pages represented in the graph are scored on content, and a set of pages is selected, the selected set of pages having scores greater than a first predetermined threshold. The selected set of pages is scored on connectivity, and a subset of the set of pages that have scores greater than a second predetermined threshold are selected as related pages.
1. A method for identifying related pages from a plurality of pages in a linked database, comprising the steps of:
selecting an initial page from the plurality of pages;
identifying a plurality of pages linked to the initial page;
identifying a plurality of stop URLs, said determination including an analysis of incoming and outgoing links associated with the stop URL, wherein the stop URL is a URL frequency referenced by another plurality of pages, content of a web pages associated with the URL is general in nature and is unrelated to one or more topics of the initial page;
representing the initial page and the plurality of linked pages as a graph of undirected nodes and edges in a memory, the nodes excluding the one or more stop URLs;
repeatedly scoring the initial page and the pages linked to the initial page, where the scoring is based on connectivity of the pages; and
selecting a subset of the pages scored on connectivity that have scores greater than a first predetermined threshold as the related pages of the linked database; and
storing the selected subset of pages in a computerized memory device.
2. The method of claim 1 further including: scoring the pages represented in the graph on content of the pages; and
selecting the subset of the pages scored on content that have scores greater than a second predetermined threshold.
3. The method of claim 2 wherein the pages are scored on content by measuring the similarity of the pages to a topic.
4. The method of claim 3 wherein the topic is extracted from the initial page.
5. The method of claim 3 wherein the topic is extracted from the pages represented in the graph.
6. The method of claim 2 including removing any nodes from the graph that have scores higher than a third predetermined threshold.
7. The method of claim 6 wherein the third predetermined threshold is larger than ninety percent of the score.
8. The method of claim 6 wherein the third predetermined threshold is at least three times larger than a next highest scoring node.
9. The method of claim 1 wherein the initial page is selected by specifying an address of the page.
10. The method of claim 1 wherein the initial page is selected by a user interface.
11. The method of claim 1 wherein pages linked in any direction to the initial page are represented in the graph.
12. The method of claim 11 wherein the pages represented in the graph are linked to the initial page by a predetermined number of links.
13. The method of claim 11 wherein each page represented in the graph depends on a path from each page to the initial page, the path including the length of the path and the direction of edges on the path.
14. The method of claim 11 wherein the pages represented in the graph as nodes are linked to the node representing the initial page by a number of edges that is determined dynamically.
15. The method of claim 1 performed in a client computer.
16. The method of claim 1 performed in a server computer.
17. The method of claim 1 , wherein connectivity of the pages is determined by the number of edges that need to be traversed on the graph to reach from the initial page to one of the pages linked to the initial page.