Automated web page accessing
An apparatus for accessing web pages includes: a communication interface; and a processor communicatively coupled to the communication interface and configured to: receive a web crawler request, from the communication interface, to access a specified web page; select a first selected proxy server from among a plurality of proxy servers; send, via the communication interface, a first access request to the first selected proxy server requesting access to the specified web page; receive, via the communication interface, a response from the selected proxy server corresponding to the first access request; analyze the response from the selected proxy server; select a second selected proxy server from among the plurality of proxy servers in response to determining from the response that the first access request was denied, and send, via the communication interface, a second access request to the second selected proxy server requesting access to the specified web page.
1. An apparatus for accessing web pages, the apparatus comprising:
a communication interface; and
a processor communicatively coupled to the communication interface and configured to:
receive a web crawler request, from the communication interface, to access a specified web page;
select a first selected proxy server from among a plurality of proxy servers;
send, via the communication interface, a first access request to the first selected proxy server requesting access to the specified web page;
receive, via the communication interface, a response from the first selected proxy server corresponding to the first access request;
analyze the response from the first selected proxy server;
select a second selected proxy server from among the plurality of proxy servers in response to determining from the response that the first access request was denied; and
send, via the communication interface, a second access request to the second selected proxy server requesting access to the specified web page.
2. The apparatus of claim 1 , wherein the processor is configured to schedule a third access request requesting access to the specified web page for later transmission to one of the plurality of proxy servers of the plurality of proxy servers in response to a threshold amount of failed access to the specified web page.
3. The apparatus of claim 2 , wherein the threshold amount of failed access is a threshold number of access requests being denied access to the specified web page.
4. The apparatus of claim 2 , wherein the threshold amount of failed access is a threshold amount of time since the first access request is sent.
5. The apparatus of claim 2 , wherein to schedule the third access request the processor is configured to place the third access request in a queue of web page requests.
6. The apparatus of claim 1 , wherein to select the first selected proxy server the processor is configured to select randomly from among the plurality of proxy servers.
7. The apparatus of claim 1 , wherein the processor is configured to establish persistent connections to a web crawler and the plurality of proxy servers via the communication interface.
8. The apparatus of claim 1 , wherein the first access request and the second access request are configured to request access to all of the specified web page.
9. A non-transitory, processor-readable storage medium comprising processor-readable instructions stored thereon configured to cause a processor to:
select a first selected proxy server from among a plurality of proxy servers in response to receipt of a web crawler request, from a communication interface, to access a specified web page;
send, via the communication interface, a first access request to the first selected proxy server requesting access to the specified web page;
analyze a response from the first selected proxy server corresponding to the first access request;
select a second selected proxy server from among the plurality of proxy servers in response to determining from the response that the first access request was denied; and
send, via the communication interface, a second access request to the second selected proxy server requesting access to the specified web page.
10. The storage medium of claim 9 , wherein the instructions are configured to cause the processor to schedule a third access request requesting access to the specified web page for later transmission to one of the plurality of proxy servers of the plurality of proxy servers in response to a threshold amount of failed access to the specified web page.
11. The storage medium of claim 10 , wherein the threshold amount of failed access is a threshold number of access requests being denied access to the specified web page.
12. The storage medium of claim 10 , wherein the threshold amount of failed access is a threshold amount of time since the first access request is sent.
13. The storage medium of claim 10 , wherein to schedule the third access request the instructions are configured to cause the processor to place the third access request in a queue of web page requests.
14. The storage medium of claim 9 , wherein to select the first selected proxy server the instructions are configured to cause the processor to select randomly from among the plurality of proxy servers.
15. A method for accessing a web page, the method comprising:
receiving a web crawler request, from a web crawler via a communication interface, to access a specified web page;
selecting a first selected proxy server from among a plurality of proxy servers;
sending, via the communication interface, a first access request to the first selected proxy server requesting access to the specified web page;
receiving, via the communication interface, a response from the first selected proxy server corresponding to the first access request;
analyzing the response from the first selected proxy server;
selecting a second selected proxy server from among the plurality of proxy servers in response to determining from the response that the first access request was denied; and
sending, via the communication interface, a second access request to the second selected proxy server requesting access to the specified web page.
16. The method of claim 15 , wherein the method is for accessing web pages from a crawler adapter that includes the communication interface, the method further comprising establishing always-on connections between the crawler adapter and the plurality of proxy servers.
17. The method of claim 15 , further comprising scheduling a third access request requesting access to the specified web page for later transmission to one of the plurality of proxy servers of the plurality of proxy servers in response to a threshold amount of failed access to the specified web page.
18. The method of claim 17 , wherein the threshold amount of failed access is a threshold number of access requests being denied access to the specified web page or a threshold amount of time since the first access request is sent.
19. The method of claim 17 , wherein scheduling the third access request comprises placing the third access request in a queue of web page requests.
20. The method of claim 15 , wherein selecting the first selected proxy server comprises randomly selecting the first selected proxy server from among the plurality of proxy servers.