Crawling pages is disclosed. Pages are crawled up to a target number of pages. Additional pages, that have an importance that is equal to or greater than an importance threshold, are crawled beyond the target number of pages. In some embodiments, pages having an importance less than an importance threshold are deleted.
1. A method for crawling and indexing items on an intranet using a search system, the method comprising:
maintaining a table including item locators and an importance rank for each of the item locators;
crawling items on an intranet using a search system;
adding entries in an index for each of the crawled items;
during crawling by the search system, discovering new item locators and adding the new item locators to the table;
crawling new items on the intranet associated with the new item locators using the search system; and
adding new entries in the index for the new items until a configurable number of entries in the index is reached.
2. The method as in claim 1 further comprising deleting entries in the index based on the importance rank.
3. The method as in claim 1 further comprising:
updating the importance rank for the item locators in the table; and
deleting entries in the index based on the updated importance rank.
4. The method as in claim 1 further comprising adding the new entries in the index based on an importance rank.
5. The method as in claim 1 wherein the configurable number of entries in the index is a number associated with a license limiting the number of entries in the index.
6. The method as in claim 1 further comprising stopping crawling when the configurable number of entries in the index is reached.
7. The method as in claim 1 wherein the item locators include uniform resource locators (URLs).
8. The method as in claim 1 wherein the crawled items include documents.
9. The method as in claim 1 wherein the crawled items include multimedia files.
10. A system for crawling and indexing items on an intranet comprising:
one or more processors of a search system, the one or more processors being configured to:
maintain a table including item locators and an importance rank for each of the item locators;
crawl items on an intranet;
add entries in an index for each of the crawled items;
during crawling, discover new item locators and add the new item locators to the table;
crawl new items on the intranet associated with the new item locators; and
add new entries in the index for the new items until a configurable number of entries in the index is reached.
11. The system of claim 10 wherein the one or more processors are configured to delete entries in the index based on the importance rank.
12. The system of claim 10 wherein the one or more processors are configured to:
update the importance rank for the item locators in the table; and
delete entries in the index based on the updated importance rank.
13. The system of claim 10 wherein the one or more processor are configured to add the new entries in the index based on an importance rank.
14. The system of claim 10 wherein the configurable number of entries in the index is a number associated with a license limiting the number of entries in the index.
15. The system of claim 10 wherein the one or more processors are configured to stop crawling when the configurable number of entries in the index is reached.
16. The system of claim 10 wherein the item locators include uniform resource locators (URLs).
17. The system of claim 10 wherein the crawled items include documents.
18. The system of claim 10 wherein the crawled items include multimedia files.
19. A non-transitory computer-readable storage device comprising instructions for crawling and indexing items on an intranet that, when executed, cause one or more processors of a search system to perform the actions of:
maintaining a table including item locators and an importance rank for each of the item locators;
crawling items on an intranet;
adding entries in an index for each of the crawled items;
during crawling, discovering new item locators and adding the new item locators to the table;
crawling new items on the intranet associated with the new item locators; and
adding new entries in the index for the new items until a configurable number of entries in the index is reached.
20. The non-transitory computer-readable storage device of claim 19 further comprising instructions that, when executed, cause the one or more processors to perform the action of deleting entries in the index based on the importance rank.
21. The non-transitory computer-readable storage device of claim 19 further comprising instructions that, when executed, cause the one or more processors to perform the actions of:
updating the importance rank for the item locators in the table; and
deleting entries in the index based on the updated importance rank.
22. The non-transitory computer-readable storage device of claim 19 further comprising instructions that, when executed, cause the one or more processors to perform the action of adding the new entries in the index based on an importance rank.
23. The non-transitory computer-readable storage device of claim 19 wherein the configurable number of entries in the index is a number associated with a license limiting the number of entries in the index.
24. The non-transitory computer-readable storage device of claim 19 further comprising instructions that, when executed, cause the one or more processors to perform the action of stopping crawling when the configurable number of entries in the index is reached.
25. The non-transitory computer-readable storage device of claim 19 wherein the item locators include uniform resource locators (URLs).
26. The non-transitory computer-readable storage device of claim 19 wherein the crawled items include documents.
27. The non-transitory computer-readable storage device of claim 19 wherein the crawled items include multimedia files.