System and method for crawl policy management utilizing IP address and IP address range
View Patent ↗The present invention relates to a method for configuring a policy management protocol for a web crawler, the method further comprising the steps of determining a web space that is to be crawled by a web crawler, wherein the web space is comprised of an IP address and/or a range of IP addresses, and determining additional hostnames that are associated with the IP address and/ range of IP addresses. The method further comprises the steps of configuring the web crawler to crawl the IP address and/ range of IP addresses, and determine additional hostnames that are associated with the IP address or range of IP addresses, and performing a web crawling function upon the determined additional hostnames by the web crawler.
1. A computer-implemented method for configuring a policy management protocol for a web crawler, a computer executing computer code to perform the method comprising:
determining a web space that is to be crawled by a web crawler, wherein the web space is comprised of at least one IP address and at least one range of IP addresses;
storing the at least one IP address and the at least one range of IP addresses within a URL frontier;
determining additional hostnames that are associated with the at least one IP address by performing a reverse DNS lookup of the at least one IP address;
configuring the web crawler to crawl the at least one IP address or at least one range of IP addresses, and determine additional hostnames that are associated with the at least one IP address or at least one range of IP addresses; and
performing a web crawling function upon the determined the at least one IP address, the at least one range of IP addresses, and the determined additional hostnames by the web crawler;
wherein determining the web space further comprises determining at least one IP address and one range of IP addresses that the web crawler is restricted to crawling;
wherein the IP address and range of IP addresses are associated with geographic locations;
wherein the IP address and range of IP addresses are associated with a predetermined taxonomy as defined by a regional Internet registry;
wherein performing web crawling includes evaluating a link in a crawled website to determine if the link belongs to the web space, the evaluating including:
determining that the host identification segment of the link is an IP address and adding the link to the URL frontier if the link IP address matches the at least one IP address or at least one range of IP addresses of the web space;
determining that the host identification segment of the link is not an IP address and adding the link to the URL frontier if link fulfills host/prefix rules for the web space;
determining that the host identification segment of the link is not an IP address, determining that the link does not fulfill host/prefix rules for the web space; performing a DNS lookup operation to obtain an IP address for the link, and adding the link to the URL frontier if the link IP address matches the at least one IP address or at least one range of IP addresses of the web space.