Direct leg access for proxy web scraping
Embodiments disclose a system that allows for improved generation of web requests for scraping that, because of the nature of the requests and time and manner they are sent out, appear more organic, as in human generated, than conventional automated scraping systems. The system then manages how a client request to scrape a target website is made to the site, masking the request in a manner that makes it appear to the Web server as if the request is not generated by an automated system. In this way, by appearing more organic, Web servers may be less likely to block requests from the disclosed system or may take longer to block requests from the disclosed system. By avoiding Web servers blocking requests and extending the lifetime of IP proxies before they are blocked, embodiments can use a limited IP proxy address space more efficiently.
1. A computer-implemented method for securing a web scraping system, comprising:
(a) receiving, at a load balancer, an API request to scrape content from a target website from a client computing device, wherein the load balancer is identified based on a hostname in the API request;
(b) generating a token indicating an identity of a client associated with the API request;
(c) identifying a server configured to initiate a scraping process on the web scraping system based on the hostname in the API request;
(d) transmitting the API request along with the token to the server;
at the server configured to initiate the scraping process on the web scraping system:
(e) authorizing the client using the token; and
(f) causing the web scraping system to scrape the target website in response to the authorization.
2. The method of claim 1 , further comprising:
(g) passing the API request between a plurality of servers, each configured to perform a function of the web scraping system, the server configured to initiate the web scraping system being included in the plurality of servers;
at each of the plurality of servers:
(h) analyzing the token to determine whether the client is authorized to conduct the function performed by the server; and
(i) when the client is authorized to conduct the function, performing the function.
3. The method of claim 2 , wherein the plurality of servers includes a server configured to service API requests formatted as a web proxy request.
4. The method of claim 1 , wherein the hostname indicates that the API request is to be transmitted to a server configured to service synchronous API requests, wherein the server configured to service synchronous API requests leaves a connection between the web scraping system and the client computing device open while the web scraping system scrapes the target website.
5. The method of claim 1 , wherein the hostname indicates that the API request is to be transmitted to a server configured to service asynchronous API requests, wherein the server configured to service asynchronous API requests closes a connection between the web scraping system and the client computing device before the web scraping system scrapes the target website.
6. The method of claim 1 , wherein the generating (b) comprises generating the token to include a role of the client.
7. The method of claim 1 , wherein the generating (b) comprises generating the token to include a digital signature that cryptographically guarantees that the identity of the client has not been tampered with.
8. The method of claim 1 , wherein the API request is a first API request, and the token is a first token, further comprising:
(g) receiving a second API request asking to retrieve content that the web scraping system has previously scraped from the target website;
(h) generating a second token indicating an identity of a client associated with the second API request;
(i) determining whether the first and second tokens indicate that the first and second API requests came from the client; and
(j) when the first and second tokens indicate that the first and second API requests came from the client, returning the scraped content in response to the second API request.
9. The method of claim 8 , further comprising, when the first and second tokens do not indicate that the first and second API requests came from the client, refusing to return the scraped content.
10. The method of claim 1 , wherein the load balancer selects the server from a plurality of parallel servers.
11. A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations, the operations comprising:
(a) receiving, at a load balancer, an API request to scrape content from a target website from a client computing device, wherein the load balancer is identified based on a hostname in the API request;
(b) generating a token indicating an identity of a client associated with the API request;
(c) identifying a server configured to initiate a scraping process on a web scraping system based on the hostname in the API request;
(d) transmitting the API request along with the token to the server;
at the server configured to initiate the scraping process on the web scraping system:
(e) authorizing the client using the token; and
(f) causing the web scraping system to scrape the target website in response to the authorization.
12. The device of claim 11 , the operations further comprising:
(g) passing the API request between a plurality of servers, each configured to perform a function of the web scraping system, the server configured to initiate the web scraping system being included in the plurality of servers;
at each of the plurality of servers:
(h) analyzing the token to determine whether the client is authorized to conduct the function performed by the server; and
(i) when the client is authorized to conduct the function, performing the function.
13. The device of claim 12 , wherein the plurality of servers includes a server configured to service API requests formatted as a web proxy request.
14. The device of claim 11 , wherein the hostname indicates that the API request is to be transmitted to a server configured to service synchronous API requests, wherein the server configured to service synchronous API requests leaves a connection between the web scraping system and the client computing device open while the web scraping system scrapes the target website.
15. The device of claim 11 , wherein the hostname indicates that the API request is to be transmitted to a server configured to service asynchronous API requests, wherein the server configured to service asynchronous API requests closes a connection between the web scraping system and the client computing device before the web scraping system scrapes the target website.
16. The device of claim 11 , wherein the generating (b) comprises generating the token to include a role of the client.
17. The device of claim 11 , wherein the generating (b) comprises generating the token to include a digital signature that cryptographically guarantees that the identity of the client has not been tampered with.
18. The device of claim 11 , wherein the API request is a first API request, and the token is a first token, the operations further comprising:
(g) receiving a second API request asking to retrieve content that the web scraping system has previously scraped from the target website;
(g) generating a second token indicating an identity of a client associated with the second API request;
(h) determining whether the first and second tokens indicate that the first and second requests came from the client; and
(e) when the first and second tokens indicate that the first and second requests came from the client, returning the scraped content in response to the second API request.
19. The device of claim 18 , the operations further comprising, when the first and second tokens do not indicate that the first and second requests came from the client, refusing to return the scraped content.
20. The device of claim 11 , wherein the load balancer selects the server from a plurality of parallel servers.