Method and system for triggering web crawling based on registry data

1. A computer-implemented method of triggering crawling of a domain, the method comprising:

receiving information related to the domain from a registrar;

processing the information related to the domain;

storing the processed information in a registry zone file;

forming a list of registry data based on the processed information, wherein the list of registry data comprises a subset of the registry zone file;

crawling one or more domains in the list of registry data;

establishing a threshold value for domain resolution requests;

determining that a number of domain resolution requests for the domain in a predetermined time period is greater than the threshold value; and

adding the domain to the list of registry data.

2. The method of claim 1 wherein the registrar is one of a plurality of registrars associated with a shared registration system.

3. The method of claim 1 wherein the information related to the domain is received in accordance with Extensible Provisioning Protocol (EPP).

4. The method of claim 1 wherein the information related to the domain is related to addition, deletion, or transfer of the domain.

5. The method of claim 1 wherein the registry zone file comprises a nameserver for the domain and an IP address for the nameserver.

6. The method of claim 1 further comprising providing the list of registry data as an accessible resource.

7. The method of claim 6 wherein providing the list of registry data as an accessible resource comprises storing the list of registry data on a file transfer protocol (FTP) server accessible to third parties.

8. The method of claim 1 further comprising sorting the list of registry data based on a frequency of the domain resolution requests.

9. A computer-implemented method of providing domain information to a web crawler, the method comprising:

receiving information related to a first domain from a registrar during a predetermined time period;

processing the information related to the domain;

storing the information that is processed in a registry zone file;

forming a list of registry data based on the information that is processed;

establishing a threshold value for domain resolution requests;

determining that a number of resolution requests for a second domain in the predetermined time period is greater than the threshold value;

adding information related to the second domain to the list of registry data; and

making the list of registry data accessible to the web crawler.

10. The method of claim 9 wherein the registrar is one of a plurality of registrars associated with a shared registration system.

11. The method of claim 9 wherein the information related to the domain is received in accordance with Extensible Provisioning Protocol (EPP).

12. The method of claim 9 wherein the domain information is related to addition, deletion, or transfer of at least one of the first domain or the second domain.

13. The method of claim 9 wherein the list of registry data is a subset of the registry zone file.

14. The method of claim 9 wherein making the list of registry data accessible to a web crawler comprises storing the list of registry data on an FTP a file transfer protocol (FTP) server accessible to third parties.