Token-based authentication for a proxy web scraping service
Embodiments disclose a system that allows for improved generation of web requests for scraping that, because of the nature of the requests and time and manner they are sent out, appear more organic, as in human generated, than conventional automated scraping systems. The system then manages how a client request to scrape a target website is made to the site, masking the request in a manner that makes it appear to the Web server as if the request is not generated by an automated system. In this way, by appearing more organic, Web servers may be less likely to block requests from the disclosed system or may take longer to block requests from the disclosed system. By avoiding Web servers blocking requests and extending the lifetime of IP proxies before they are blocked, embodiments can use a limited IP proxy address space more efficiently.
1. A computer-implemented method for securing a web scraping system, comprising:
(a) receiving, at a load balancer, an API request to scrape content from a target website from a client computing device;
(b) generating a token indicating an identity of a client associated with the API request;
(c) identifying a server configured to initiate a scraping process on the web scraping system based on a hostname in the API request;
(d) transmitting the API request along with the token to the server;
at the server configured to initiate the scraping process on the web scraping system:
(e) authorizing the client using the token; and
(f) causing the web scraping system to scrape the target website in response to the authorization.
2. The computer-implemented method of claim 1 , further comprising:
(g) passing the API request between a plurality of servers, each configured to perform a function of the web scraping system, the server configured to initiate the web scraping system being included in the plurality of servers;
at each respective server of the plurality of servers:
(h) analyzing the token to determine whether the client is authorized to conduct the function performed by the respective server; and
(i) when the client is authorized to conduct the function, performing the function.
3. The computer-implemented method of claim 2 , wherein the plurality of servers includes a server configured to service API requests formatted as a web proxy request.
4. The computer-implemented method of claim 1 , wherein the hostname indicates that the API request is to be transmitted to a server configured to service synchronous API requests, wherein the server configured to service synchronous API requests leaves a connection between the web scraping system and the client computing device open while the web scraping system scrapes the target website.
5. The computer-implemented method of claim 1 , wherein the hostname indicates that the API request is to be transmitted to a server configured to service asynchronous API requests, wherein the server configured to service asynchronous API requests closes a connection between the web scraping system and the client computing device before the web scraping system scrapes the target website.
6. The computer-implemented method of claim 1 , wherein the generating (b) comprises generating the token to include a role of the client.
7. The computer-implemented method of claim 1 , wherein the generating (b) comprises generating the token to include a digital signature that cryptographically guarantees that the identity of the client has not been tampered with.
8. The computer-implemented method of claim 1 , wherein the API request is a first API request, and the token is a first token, further comprising:
(g) receiving a second API request asking to retrieve content that the web scraping system has previously scraped from the target website;
(h) generating a second token indicating an identity of a client associated with the second API request;
(i) determining whether the first and second tokens indicate that the first and second API requests came from the client; and
(j) when the first and second tokens indicate that the first and second API requests came from the client, returning the scraped content in response to the second API request.
9. The computer-implemented method of claim 8 , further comprising, when the first and second tokens do not indicate that the first and second API requests came from the client, refusing to return the scraped content.
10. The computer-implemented method of claim 1 , wherein the load balancer selects the server from a plurality of parallel servers.
11. A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations, the operations comprising:
(a) receiving, at a load balancer, an API request to scrape content from a target website from a client computing device;
(b) generating a token indicating an identity of a client associated with the API request;
(c) identifying a server configured to initiate a scraping process on a web scraping system based on a hostname in the API request;
(d) transmitting the API request along with the token to the server;
at the server configured to initiate the scraping process on the web scraping system:
(e) authorizing the client using the token;
(f) causing the web scraping system to scrape the target website in response to the authorization.
12. The non-transitory computer-readable device of claim 11 , the operations further comprising:
(g) passing the API request between a plurality of servers, each configured to perform a function of the web scraping system, the server configured to initiate the web scraping system being included in the plurality of servers;
at each respective server of the plurality of servers:
(h) analyzing the token to determine whether the client is authorized to conduct the function performed by the respective server; and
(i) when the client is authorized to conduct the function, performing the function.
13. The non-transitory computer-readable device of claim 12 , wherein the plurality of servers includes a server configured to service API requests formatted as a web proxy request.
14. The non-transitory computer-readable device of claim 11 , wherein the hostname indicates that the API request is to be transmitted to a server configured to service synchronous API requests, wherein the server configured to service synchronous API requests leaves a connection between the web scraping system and the client computing device open while the web scraping system scrapes the target website.
15. The non-transitory computer-readable device of claim 11 , wherein the hostname indicates that the API request is to be transmitted to a server configured to service asynchronous API requests, wherein the server configured to service asynchronous API requests closes a connection between the web scraping system and the client computing device before the web scraping system scrapes the target website.
16. The non-transitory computer-readable device of claim 11 , wherein the generating (b) comprises generating the token to include a role of the client.
17. The non-transitory computer-readable device of claim 11 , wherein the generating (b) comprises generating the token to include a digital signature that cryptographically guarantees that the identity of the client has not been tampered with.
18. The non-transitory computer-readable device of claim 11 , wherein the API request is a first API request, and the token is a first token, the operations further comprising:
(g) receiving a second API request asking to retrieve content that the web scraping system has previously scraped from the target website;
(g) generating a second token indicating an identity of a client associated with the second API request;
(h) determining whether the first and second tokens indicate that the first and second requests came from the client; and
(e) when the first and second tokens indicate that the first and second API requests came from the client, returning the scraped content in response to the second API request.
19. The non-transitory computer-readable device of claim 18 , the operations further comprising, when the first and second tokens do not indicate that the first and second API requests came from the client, refusing to return the scraped content.
20. The non-transitory computer-readable device of claim 11 , wherein the load balancer selects the server from a plurality of parallel servers.