Authentication of IP source addresses

1. A method performed by a computer system having one or more processors and memory storing one or more programs for execution by the one or more processors, comprising:

accessing one or more HTTP requests being identified as a particular webcrawler;

detecting one or more IP addresses from the one or more HTTP requests;

authenticating each detected IP address by associating it with a probability estimation regarding its association with the particular webcrawler, including:

determining a first ratio of the one or more HTTP requests of a Classless Inter-Domain Routing (CIDR) block that are authenticated by round-trip DNS look-up; and

determining a second ratio of IP addresses of the CIDR block that are associated with a request to the particular webcrawler;

compiling a lookup table of each authenticated IP address in correlation with a confidence score indicating legitimacy of the particular webcrawler, the confidence score being contingent upon the probability estimation regarding association of the IP address with the particular webcrawler; and

statistically aggregating into CIDR blocks each authenticated IP address.

2. A method as recited in claim 1 further including the step of determining if an IP address of an HTTP request to a webserver is from a legitimate webcrawler by cross referencing the IP address with the compiled lookup table of authenticated IP addresses.

3. A method as recited in claim 2 wherein internet sensors are used to access the one or more HTTP requests.

4. A method as recited in claim 2 wherein authenticating each detected IP address includes filtering malicious bots that spoof user agent strings used by legitimate webcrawlers.

5. A method as recited in claim 4 wherein authenticating each detected IP address includes the use of round-trip Domain Name Server (DNS) lookup tables.

6. A method as recited in claim 2 wherein authenticating each detected IP address includes:

the use of heuristics based on density of IP addresses in CIDR blocks; and

determining whether the CIDR block size is greater than a threshold block size.

7. A method as recited in claim 4 wherein authenticating each detected IP address includes the use of an Autonomous System Number (ASN) lookups.

8. A method as recited in claim 4 wherein authenticating each detected IP address includes cross-referencing against external open sources of crawler logs.

9. A method as recited in claim 1 further including updating the CIDR blocks having the statistically aggregated IP addresses at prescribed time periods.

10. A system for authenticating legitimate webcrawler IP addresses of HTTP requests, comprising:

a memory;

a processor disposed in communication with said memory, and configured to issue a plurality of instructions stored in the memory, wherein the instructions issue signals to:

access one or more HTTP requests being identified as a particular webcrawler;

detect one or more IP addresses from the one or more HTTP requests;

authenticate each detected IP address by associating it with a probability estimation regarding its association with the particular webcrawler, including:

determine a first ratio of the one or more HTTP requests of a Classless Inter-Domain Routing (CIDR) block that are authenticated by round-trip DNS look-up; and

determine a second ratio of IP addresses of the CIDR block that are associated with a request to the particular webcrawler;

compile a lookup table of each authenticated IP address in correlation with a confidence score indicating legitimacy of the particular webcrawler, the confidence score being contingent upon the probability estimation regarding association of the IP address with the particular webcrawler; and

issue signals to statistically aggregate into CIDR blocks each authenticated IP address.

11. A system as recited in claim 10 wherein the processor is further configured to issue signals to determine if an IP address of an HTTP request to a webserver is from a legitimate webcrawler by cross referencing the IP address with the compiled lookup table of authenticated IP addresses.

12. A system as recited in claim 11 wherein internet sensors are used to access the one or more HTTP requests.

13. A system as recited in claim 11 wherein the processor is further configured to issue signals to filter malicious bots that spoof user agent strings used by legitimate webcrawlers.

14. A system as recited in claim 13 wherein each detected IP address is authenticated via the use of round-trip Domain Name Server (DNS) lookup tables.

15. A system as recited in claim 13 wherein each detected IP address is authenticated via the use of heuristics based on density of IP addresses in CIDR blocks and depends on whether the CIDR block size is greater than a threshold block size.

16. A system as recited in claim 13 wherein each detected IP address is authenticated via the use of Autonomous System Number (ASN) lookups.

17. A system as recited in claim 10 wherein the processor is further configured to issue signals to update the CIDR blocks having the statistically aggregated IP addresses at prescribed time periods.

18. A non-transitory computer readable storage medium and one or more computer programs embedded therein, the computer programs comprising instructions, which when executed by a computer system, cause the computer system to:

access one or more HTTP requests being identified as a particular webcrawler;

detect one or more IP addresses from the one or more HTTP requests;

authenticate each detected IP address by associating it with a probability estimation regarding its association with the particular webcrawler, including:

determine a first ratio of the one or more HTTP requests of a CIDR block that are authenticated by round-trip DNS look-up; and

determine a second ratio of IP addresses of the CIDR block that are associated with a request to the webcrawler; and

compile a lookup table of each authenticated IP address in correlation with a confidence score indicating legitimacy of the particular webcrawler, the confidence score being contingent upon the probability estimation regarding its association of the IP address with the particular webcrawler, wherein each authenticated IP address is statistically aggregated into CIDR blocks.