Distributed crawling of hyperlinked documents

1. A computer implemented method of crawling hyperlinked documents, comprising:

sending a request for additional links to hyperlinked documents to a link manager;

receiving a plurality of links to hyperlinked documents to be crawled, the plurality of links being selected by the link manager based on priority;

grouping the plurality of links to hyperlinked documents by host;

grouping hosts into buckets according to a number of hyperlinked documents to be crawled at each host;

sorting the hosts in each bucket based on a stall time of each host;

selecting a host from one of the buckets to crawl next according to the stall time of the host;

crawling a hyperlinked document from the selected host;

determining a retrieval time for crawling the hyperlinked document from the selected host; and

adjusting a subsequent stall time for the selected host according to the retrieval time.

2. The method of claim 1 , wherein the stall time of the host is the earliest time in which a hyperlinked document from the host should be crawled.

3. The method of claim 1 , wherein selecting a host to crawl next includes selecting a host with a stall time that is earlier than a current time.

4. The method of claim 1 , further comprising examining the buckets in descending order of the number of hyperlinked documents to be crawled at each host until a host is found with a stall time that is earlier than a current time.

5. The method of claim 1 , further comprising moving the selected host to a bucket with less hyperlinked documents to be crawled.

6. A computer-readable storage device including a plurality of instructions that, when executed by at least one processor, causes a method to be performed, the method comprising:

requesting links from a link manager;

receiving a plurality of links to hyperlinked documents to be crawled from the link manager, the plurality of links being selected by the link manager based on priority;

grouping the plurality of links to hyperlinked documents by host;

grouping hosts into buckets according to a number of hyperlinked documents to be crawled at each host;

sorting the hosts in each bucket based on a stall time of each host;

selecting a host from one of the buckets to crawl next according to the stall time of the host;

crawling a hyperlinked document from the selected host;

determining a retrieval time for crawling the hyperlinked document from the selected host; and

adjusting a subsequent stall time for the selected host according to the retrieval time.

7. The computer-readable storage device of claim 6 , wherein the computer-readable storage device includes a CD-ROM, floppy disk, tape, flash memory, system memory, or hard drive.

8. The computer-readable storage device of claim 6 wherein selecting a host from one of the buckets to crawl next includes:

selecting a host with a stall time that is earlier than a current time.

9. The computer-readable storage device of claim 6 wherein selecting a host from one of the buckets to crawl next includes:

examining the buckets in descending order of the number of hyperlinked documents to be crawled at each host until a host is found with a stall time that is earlier than a current time.

10. The computer-readable storage device of claim 6 wherein the method further comprises:

moving the selected host to a bucket with less hyperlinked documents to be crawled after crawling the hyperlinked document from the selected host.

11. A computer implemented method of crawling hyperlinked documents, comprising:

sending a request for links to hyperlinked documents to a device;

receiving a plurality of links to hyperlinked documents to be crawled from the device, the plurality of links being selected by the device based on priority;

grouping the plurality of links to hyperlinked documents by host;

grouping hosts into buckets according to a number of hyperlinked documents to be crawled at each host;

selecting a host from one of the buckets to crawl next according to a stall time of the host;

crawling a hyperlinked document from the selected host;

determining a retrieval time for retrieving the hyperlinked document from the selected host; and

adjusting subsequent stall times for the selected host according to the retrieval time.

12. The method of claim 11 , wherein the stall time of the host is the earliest time in which a hyperlinked document from the host should be crawled.

13. The method of claim 11 , wherein selecting a host to crawl next includes selecting a host with a stall time that is earlier than a current time.

14. The method of claim 11 , further comprising examining the groups in descending order of the number of hyperlinked documents to be crawled at each host until a host is found with a stall time that is earlier than a current time.

15. The method of claim 11 , wherein the hosts within each group are sorted by stall time.

16. The method of claim 11 , further comprising moving the selected host to a group with less hyperlinked documents to be crawled.

17. A computer-readable storage device including a plurality of instructions that, when executed by at least one processor, causes a method to be performed, the method comprising:

sending a request for links to hyperlinked documents to a device;

receiving a plurality of links to hyperlinked documents to be crawled from the device, the plurality of links being selected by the device based on priority;

grouping the plurality of links to hyperlinked documents by host;

grouping hosts into buckets according to a number of hyperlinked documents to be crawled at each host;

selecting a host from one of the buckets to crawl next according to a stall time of the host;

crawling a hyperlinked document from the selected host;

determining a retrieval time for crawling the hyperlinked document from the selected host; and

adjusting a subsequent stall time for the selected host according to the retrieval time.

18. The computer-readable storage device of claim 17 , wherein the computer-readable storage device includes a CD-ROM, floppy disk, tape, flash memory, system memory, or hard drive.

19. The computer-readable storage device of claim 17 wherein selecting a host from one of the buckets to crawl next includes:

selecting a host with a stall time that is earlier than a current time.

20. The computer-readable storage device of claim 17 wherein selecting a host from one of the buckets to crawl next includes:

examining the buckets in descending order of the number of hyperlinked documents to be crawled at each host until a host is found with a stall time that is earlier than a current time.

21. The computer-readable storage device of claim 17 wherein the method further comprises:

moving the selected host to a bucket with less hyperlinked documents to be crawled after crawling the hyperlinked document from the selected host.

22. A computer implemented method of crawling hyperlinked documents, comprising:

storing a plurality of links to hyperlinked documents to be crawled;

determining that more links to hyperlinked documents are desired;

sending requests to multiple link managers for more links to hyperlinked documents;

receiving additional links to hyperlinked documents from the link managers;

selecting a host to crawl next according to a stall time of the host;

crawling a hyperlinked document from the selected host;

determining a retrieval time for crawling the hyperlinked document from the selected host; and

adjusting a subsequent stall time for the selected host according to the retrieval time.

23. A computer-readable storage device including a plurality of instructions that, when executed by at least one processor, causes a method to be performed, the method comprising:

stores storing a plurality of links to hyperlinked documents to be crawled;

determines determining that more links to hyperlinked documents are desired;

sending requests to multiple link managers for more links to hyperlinked documents;

receiving additional links to hyperlinked documents from the link managers;

selecting a host to crawl next according to a stall time of the host;

crawling a hyperlinked document from the selected host;

determining a retrieval time for crawling the hyperlinked document from the selected host, and

adjusting a subsequent stall time for the selected host according to the retrieval time.

24. The computer-readable storage device of claim 23 , wherein the computer-readable storage device includes a CD-ROM, floppy disk, tape, flash memory, system memory, or hard drive.

25. A computer-implemented method comprising:

grouping links to hyperlinked documents by host, each host being associated with a stall time;

grouping hosts into buckets according to a number of hyperlinked documents to be crawled at each host;

sorting the hosts in each bucket based on the stall time of each host;

identifying a host to crawl by examining the buckets in descending order based on the number of hyperlinked documents to be crawled at each host until a host is found with a stall time that is earlier than a current time;

crawling a hyperlinked document from the identified host;

determining a retrieval time for crawling the hyperlinked document from the identified host; and

adjusting a subsequent stall time for the identified host according to the retrieval time.