Confidence scoring of device reputation based on characteristic network behavior
The technology disclosed relates to detection of anonymous proxies and bots making requests to a cloud based resource on the Internet, such as a web server or an App server. The technology can leverage one or more of: instrumentation of web pages that samples response times and other characteristics of communications by a requestor device over multiple network segments; lack of prior appearance of the requestor device across multiple, independently operated commercial web sites; and resolver usage by the requestor. These signals can be analyzed to score a requesting device's reputation. A location reported by a user device can be compared to a network characteristic determined location.
1. A method of evaluating reputation of a requestor device that makes a request to a cloud-based resource on the internet, including:
providing an initial response to the requestor device making a request to a cloud-based resource on the internet, the initial response including an instrumented web page or instructions to be processed by an application running on the requestor device, wherein the initial response includes code adapted to:
collect data regarding at least internet round trip latency between the requestor device and four or more target addresses, each target address being different from one another, and
report the internet round trip latency for the target addresses;
compiling a characteristic vector for the requestor device including at least the reported internet round trip latency for the target addresses;
scoring the characteristic vector for similarity to expected characteristics of a first reference device at a first reference IP address expected to share internet round trip latency characteristics with the requestor device; and
storing, on a non-transitory computer-readable medium, at least one reputation score, wherein the reputation score is based at least in part on the scoring the characteristic vector and correlates with a likelihood that the requestor device is a bot or is operating through an anonymous proxy server.
2. The method of claim 1 , further including:
scoring the characteristic vector using a median of estimated probability measures of characteristics in the characteristic vector taken individually.
3. The method of claim 1 , further including:
scoring the characteristic vector by combining estimated variance from the expected round trip latency characteristics for the target addresses in the characteristic vector.
4. The method of claim 1 , further including:
scoring the characteristic vector using an estimated joint probability distribution that combines at least one characteristic from each of at least four target addresses, wherein the joint probability distribution is estimated from at least 1000 samples that combine the least one characteristic from the at least four target addresses.
5. The method of claim 1 , further including:
providing the code further adapted to:
collect data regarding at least availability of target addresses from the requestor device and
report the availability of the target addresses; and
compiling in the characteristic vector a reported availability of the target addresses to the requestor device; and
scoring the characteristic vector by determining whether the availability of targets in the characteristic vector is different than expected availability of the targets to a second reference device at a second reference IP address expected to share availability status with the requestor device.
6. The method of claim 5 , further including:
scoring the characteristic vector by requiring the same availability status as expected for all targets in the characteristic vector.
7. The method of claim 1 , further including:
providing the code further adapted to:
collect data regarding at least throughput rates between the requestor device and the target addresses and
report the throughput rates for the target addresses; and
compiling in the characteristic vector a reported throughput rates between the requestor device and the target addresses; and
scoring the characteristic vector for similarity to expected characteristics of a third reference requestor device at a third reference IP address expected to share throughput rate characteristics with the requestor device.
8. The method of claim 1 , further including: scoring the characteristic vector by combining estimated variances between throughput in the characteristic vector and expected throughput rate characteristics for the target addresses.
9. The method of claim 1 , further including:
providing the code further adapted to:
collect data regarding at least connection establishment times for connections between the requestor device and the target addresses and
report the connection establishment times for the target addresses; and
compiling in the characteristic vector a reported connection establishment times between the requestor device and the requestor device; and
scoring the characteristic vector for similarity to expected characteristics of a fourth reference requestor device at a fourth reference IP address expected to share connection establishment time characteristics with the requestor device.
10. The method of claim 1 , further including:
receiving requestor device characteristics including at least an IP address, browser type and version identifiers, and operating system type and version identifiers with a request from the requestor device;
looking up in a requestor history database, that reflects requests compiled from more than 100 independently operating servers, a frequency of requests made by devices sharing the requestor device characteristics; and
scoring the requestor device characteristics for frequency and/or diversity of requests made to the independently operating servers within a predetermined recent time.
11. The method of claim 1 , further including:
scoring requestor device characteristics using logarithmic scaling of the frequency and/or diversity of the requests made by devices sharing the requestor device characteristics.
12. The method of claim 1 , further including:
providing the code further adapted to:
collect data regarding a resolver used by the requestor device to find IP addresses corresponding to fully qualified domain names and
report the resolver used by the requestor device; and
scoring the characteristic vector for matching expected resolver usage of a reference requestor device at a reference IP address expected to share resolver usage characteristics with the requestor device and producing at least one reputation score.
13. The method of claim 1 , further including:
scoring the characteristic vector by combining estimated variance from resolver usage characteristics of the requestor device.
14. A method of evaluating reputation of a requestor device that makes a request to a web site over the internet, including:
receiving requestor device characteristics including at least an internet protocol address, browser type and version identifiers, and operating system type and version identifiers with a request from the requestor device making the request to the web site;
looking up in a requestor history database, that reflects requests compiled from more than 100 independently operating servers, a frequency of requests made by devices sharing the requestor device characteristics;
scoring the requestor device characteristics for frequency and/or diversity of requests made to the independently operating servers within a predetermined recent time; and
storing, on a non-transitory computer-readable medium, the scored requestor device characteristics, wherein the scored requestor device characteristics indicate whether the requestor device is a bot or is operating through an anonymous proxy server.
15. The method of claim 14 , further including:
scoring the requestor device characteristics using logarithmic scaling of the frequency and/or diversity of the requests made by devices sharing the requestor device characteristics.
16. The method of claim 14 , further comprising:
providing an initial response to a requestor device that includes an instrumented web page or instructions to be processed by an application running on the requestor device, wherein the initial response includes code adapted to:
collect data regarding at least availability of target addresses from the requestor device and
report the availability of the target addresses; and
compiling in a characteristic vector a reported availability of the target addresses to the requestor device; and
scoring the characteristic vector by determining whether the availability of targets in the characteristic vector is different than expected availability of a reference device at a reference internet protocol address expected to share availability status with the requestor device.
17. A method of evaluating reputation of a requestor device that makes a request to a web site, including:
responsive to a request from the requestor device making a request to the web site, providing an initial response to a requestor device that includes an instrumented web page or instructions to be processed by an application running on the requestor device, wherein the initial response includes code adapted to:
collect and compile in a characteristic vector data regarding a resolver used by the requestor device to find IP addresses corresponding to fully qualified domain names and
report the resolver used by the requestor device; and
scoring the characteristic vector for matching expected resolver usage of a reference requestor device at a reference internet protocol address expected to share resolver usage characteristics with the requestor device; and
storing, on a non-transitory computer-readable medium, at least one reputation score wherein the at least one reputation score correlates with a likelihood that the requestor device is a bot or is operating through an anonymous proxy server.
18. The method of claim 17 , further including:
scoring the characteristic vector by combining estimated variance from the resolver usage characteristics of the requestor device.
19. The method of claim 17 , further comprising:
providing the code further adapted to:
collect data regarding at least availability of target addresses from the requestor device and
report the availability of the target addresses; and
compiling in the characteristic vector a reported availability of the target addresses to the requestor device; and
scoring the characteristic vector by determining whether the availability of targets in the characteristic vector is different than expected availability of a reference device at a reference internet protocol address expected to share availability status with the requestor device.