Method for ranking web page search results

1. A method for including a document in an index in a hyperlinked environment, comprising the acts of:

receiving a document to be processed;

locating a set of documents that include hyperlinks to the document;

retrieving anchortext associated with at least one of the hyperlinks;

parsing the anchortext into one or more tokens;

for each token:

determining a weight for the token,

determining whether the weight assigned to the token exceeds a threshold token weight; and

indexing the document under the token, if the token weight assigned to the token exceeds the threshold token weight, wherein the indexing act includes assigning to the token a location within the index corresponding to part of the page being indexed that is allocated for tokens having a higher degree of importance than other tokens in the same page.

2. The method of claim 1 , wherein the indexing act comprises including in the index an indication of weight for each token under which each page is indexed.

3. The method of claim 1 , wherein the indexing act comprises assigning to the token a location within the index that corresponds to the beginning of the page being indexed.

4. The method of claim 1 , wherein the weight of each token is based on its frequency of occurrence within the index.

5. The method of 1 , wherein the act of determining a weight comprises: determining a first frequency at which the anchortext appears in the index; determining a second frequency at which each token derived from the anchortext appears in the index; and assigning a weight to the token, wherein the weight is a function of the first and second frequencies.

6. The method of claim 5 , further comprising dividing the first frequency by the second frequency to produce a weight quotient; and multiplying the weight quotient by an anchor text count for the token.

7. The method of 5 further comprising determining a normalized weight for each token.

8. A program product embedded in a computerized machine-readable medium for including a document in an index in a hyperlinked environment, comprising the instructions for:

receiving a document to be processed;

locating a set of documents that include hyperlinks to the document;

retrieving anchortext associated with each hyperlink;

parsing the anchortext into one or more tokens;

and program instructions for each token comprising instructions for:

determining a weight for the token,

determining whether the weight assigned to the token exceeds a threshold token weight; and

indexing the document under the token, if the token weight assigned to the token exceeds the threshold token weight, wherein the indexing includes assigning to the token a location within the index corresponding to part of the page being indexed that is allocated for tokens having a higher degree of importance than other tokens in the same page.

9. The computer program product of claim 8 wherein the indexing instruction comprises including in the index an indication of weight for each token under which each page is indexed.

10. The computer program product of claim 8 , wherein the weight of each token is based on its frequency of occurrence within the index.

11. The computer program product of claim 8 , wherein the indexing act comprises assigning to the token a location within the index that corresponds to the beginning of the page being indexed.

12. The computer program product of claim 8 , wherein the instruction of determining a weight comprises:

determining a first frequency at which the anchortext appears in the index;

determining a second frequency at which each token derived from the anchortext appears in the index; and

assigning a weight to the token, wherein the weight is a function of the first and second frequencies.

13. The program product of claim 8 , further comprising the instruction of determining a normalized weight for each token.

14. A computer system for indexing a document in a hyperlinked environment, comprising:

a receiver for receiving a document to be processed for inclusion in an index of documents;

a module for locating a set of documents that include hyperlinks to the document;

a module for retrieving anchortext associated with each hyperlink;

a parsing module for parsing the anchortext into one or more tokens;

a module for:

determining a weight for the token,

determining whether the weight assigned to the token exceeds a threshold token weight; and

indexing the document under the token, if the token weight assigned to the token exceeds the threshold token weight, wherein indexing includes assigning to the token a location within the index corresponding to part of the page being indexed that is allocated for tokens having a higher degree of importance than other tokens in the same page.