Using network traffic logs for search engine index updates
A method and apparatus for using network traffic logs for search enhancement is disclosed. According to one embodiment, network usage is tracked by generating log files. These log files among other things indicate the frequency web pages are referenced and modified. These log files or information from these log files can then be used to improve document ranking, improve web crawling, determine tiers in a multi-tiered index, determine where to insert a document in a multi-tiered index, determine link weights, and update a search engine index.
1. A method comprising:
determining whether an information source requested by a user has been modified within a specified length of time based on a Last-Modified field of a Hypertext Transfer Protocol (HTTP) header of a message associated with the information source;
in response to determining that the information source has been modified within the specified length of time, creating an entry in a modified information source log that identifies the information source;
retrieving information from the information source based on the modified information source log; and
updating a search engine index based on the information;
wherein the method is performed by one or more computing devices.
2. The method of claim 1 , wherein the information source is a web page.
3. The method of claim 1 , further comprising:
increasing a frequency with which the information source is crawled to update the search engine index based on the modified information source log.
4. A non-transitory computer-readable storage medium storing one or more sequences of instructions that when executed by one or more processors causes the one or more processors to perform the steps of:
determining whether an information source requested by a user has been modified within a specified length of time based on a Last-Modified field of a Hypertext Transfer Protocol (HTTP) header of a message associated with the information source;
in response to determining that the information source has been modified within the specified length of time, creating an entry in a modified information source log that identifies the information source;
retrieving information from the information source based on the modified information source log; and
updating a search engine index based on the information.
5. The non-transitory computer-readable storage medium of claim 4 , wherein the information source is a web page.
6. The non-transitory computer-readable storage medium of claim 4 , further comprising instructions for:
increasing a frequency with which the information source is crawled to update the search engine index based on the modified information source log.
7. The method of claim 1 , further comprising sending the modified information source log to a search engine that maintains the search engine index.
8. The method of claim 7 , wherein the sending is performed periodically.
9. The method of claim 1 , further comprising:
forwarding the request to a content server hosting the information source;
receiving a response from the content server that specifies when the information source was last modified.
10. The method of claim 1 , wherein the steps of determining whether the information source requested by the user has been modified in the specified period of time and creating the entry in the modified information source log is performed by a traffic server.
11. The non-transitory computer-readable medium of claim 4 , further comprising instructions for sending the modified information source log to a search engine that maintains the search engine index.
12. The non-transitory computer-readable medium of claim 11 , wherein the sending is performed periodically.
13. The non-transitory computer-readable medium of claim 4 , further comprising instructions for:
forwarding the request to a content server hosting the information source;
receiving a response from the content server that specifies when the information source was last modified.
14. The non-transitory computer-readable medium of claim 4 , wherein determining whether the information source requested by the user has been modified in the specified period of time and creating the entry in the modified information source log is performed by a traffic server.