Client-specific proxy pool management, and applications thereof
Provided herein are system, apparatus, device, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for proxy pool management on a client-specific basis. Each client has his own configuration for a proxy pool. While multiple clients may share a proxy pool and pool indexes, each client has his own cursor designating its own position within the pool index. When a request is made by a client, the proxy is selected at the position of the client's cursor. By having client-specific cursors, each client can take advantage of a greater level of unique IP addresses as offered by the pool.
1 . A method for managing proxy pools, comprising:
in response to a first request to scrape data from a first target server, the first request from a first client:
(a) identifying a proxy pool index associated with the first client, the proxy pool index specifying an ordered plurality of exit proxies satisfying criteria of the first client;
(b) retrieving a first cursor of the first client, the first cursor defining a first position of the first client within the ordered plurality of exit proxies;
(c) advancing the first cursor to a first next position in the ordered plurality of exit proxies;
(d) selecting, from the ordered plurality of exit proxies, a first exit proxy at the first next position according to the first cursor;
(e) transmitting the first request to the first target server via the selected first exit proxy;
in response to a second request to scrape data from a second target server, the second request from a second client associated with the proxy pool index:
(f) retrieving a second cursor of the second client, the second cursor defining a second position of the second client within the ordered plurality of exit proxies and the second cursor being different than the first cursor;
(g) advancing the second cursor to a second next position in the ordered plurality of exit proxies;
(h) selecting, from the ordered plurality of exit proxies, a second exit proxy at the second next position according to the second cursor; and
(i) transmitting the second request to the second target server via the selected second exit proxy.
2 . The method of claim 1 , wherein the exit proxies in the ordered plurality are selected to satisfy a request option of the first and second client.
3 . The method of claim 2 , wherein the request option specifies at least one of a region, autonomous system number, priority with which to treat the traffic, IP version, or a database specifying characteristics of IP addresses.
4 . The method of claim 1 , wherein the first request specifies a first option specifying how the first exit proxy is to be selected, and the second request specifies a second option specifying how the second exit proxy is to be selected, the first and second options being different.
5 . The method of claim 4 , wherein the selecting (d) comprises selecting the first exit proxy as a first next exit proxy in the ordered plurality of exit proxies that satisfies the first option, and wherein the selecting (d) comprises selecting the second exit proxy as a second next exit proxy in the ordered plurality of exit proxies that satisfies the second option.
6 . The method of claim 1 , wherein the advancing (c) and advancing (g) occur independently of each other.
7 . The method of claim 1 , further comprising, in response to a third request to scrape data from a third target server, the third request from the first client:
determining that no valid cursor exists for the first client;
selecting a random proxy from the proxy pool index; and
creating the first cursor according to a position of the random proxy in the ordered plurality of exit proxies.
8 . The method of claim 1 , wherein the ordered plurality of exit proxies is a circular doubly linked list.
9 . The method of claim 1 , wherein the selected first and second exit proxies are the same.
10 . The method of claim 1 , wherein the selected first and second exit proxies each belong to a plurality of proxy pools.
11 . The method of claim 1 , wherein the selecting (c) comprises selecting the first exit proxy such that the first exit proxy is a next exit proxy after the first cursor that is not on a blocked IP address list.
12 . The method of claim 1 , wherein the identifying the proxy pool index (a) comprises selecting, from a data store, the proxy pool index such that the proxy pool index satisfies criteria specified in the first request, wherein the second request also specifies the criteria of the proxy pool index.
13 . The method of claim 12 , wherein the identifying the proxy pool index (a) further comprises, when no proxy pool index is present in the data store that satisfies the criteria specified in the first request, generating the proxy pool index and storing the proxy pool index in the data store.
14 . The method of claim 12 , further comprising:
(k) determining that no request specifying the criteria has been received in a particular time period; and
(l) in response to the determination in (k), removing the proxy pool index from the data store.
15 . A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:
in response to a first request to scrape data from a first target server, the first request from a first client:
(a) identifying a proxy pool index associated with the first client, the proxy pool index comprising an ordered plurality of exit proxies satisfying criteria of the first client;
(b) retrieving a first cursor of the first client, the first cursor defining a first position of the first client within the ordered plurality of exit proxies;
(c) advancing the first cursor to a first next position in the ordered plurality of exit proxies;
(d) selecting, from the ordered plurality of exit proxies, a first exit proxy at the first next position according to the first cursor;
(e) transmitting the first request to the first target server via the selected first exit proxy;
in response to a second request to scrape data from a second target server, the second request from a second client associated with the proxy pool index:
(f) retrieving a second cursor of the second client, the second cursor defining a second position of the second client within the ordered plurality of exit proxies and the second cursor being different than the first cursor;
(g) advancing the second cursor to a second next position in the ordered plurality of exit proxies;
(h) selecting, from the ordered plurality of exit proxies, a second exit proxy at the second next position according to the second cursor; and
(i) transmitting the second request to the second target server via the selected second exit proxy.
16 . The non-transitory computer-readable device of claim 15 , wherein the exit proxies in the ordered plurality are selected to satisfy a configuration option of the first and second client.
17 . The non-transitory computer-readable device of claim 16 , wherein the configuration option specifies at least one of a country, autonomous system number, priority with which to treat the traffic, IP version, or a database specifying characteristics of IP addresses.
18 . The non-transitory computer-readable device of claim 15 , wherein the first request specifies a first option specifying how the first exit proxy is to be selected, and the second request specifies a second option specifying how the second exit proxy is to be selected, the first and second options being different.
19 . The non-transitory computer-readable device of claim 18 , wherein the selecting (d) comprises selecting the first exit proxy as a first next exit proxy in the ordered plurality of exit proxies that satisfies the first option, and wherein the selecting (h) comprises selecting the second exit proxy as a second next exit proxy in the ordered plurality of exit proxies that satisfies the second option.
20 . The non-transitory computer-readable device of claim 15 , wherein the advancing (c) and advancing (g) occur independently of each other.
21 . The non-transitory computer-readable device of claim 15 , further comprising, in response to a third request to scrape data from a third target server, the third request from the first client:
determining that no valid cursor exists for the first client;
selecting a random exit proxy from the proxy pool index; and
creating the first cursor according to a position of the random exit proxy in the ordered plurality of exit proxies.
22 . The non-transitory computer-readable device of claim 15 , wherein the ordered plurality of exit proxies is a circular doubly linked list.
23 . The non-transitory computer-readable device of claim 15 , wherein the selected first and second exit proxies each belong to a plurality of proxy pools.
24 . The non-transitory computer-readable device of claim 15 , wherein the identifying the proxy pool index (a) comprises selecting the proxy pool index such that the pool index satisfies criteria specified in the first request, wherein the second request also specifies the criteria of the proxy pool index.
25 . A method for managing proxy pools, comprising:
in response to a first request to scrape data from a first target server, the first request from a first client:
(a) identifying a proxy pool index associated with the first client, the proxy pool index comprising an ordered plurality of exit proxies satisfying criteria of the first client;
(b) retrieving a first cursor of the first client, the first cursor defining a first position of the first client within the ordered plurality of exit proxies;
(c) selecting, from the ordered plurality of exit proxies, a first exit proxy at the first position according to the first cursor;
(d) transmitting the first request to the first target server via the selected first exit proxy;
(e) advancing the first cursor to a first next position in the ordered plurality of exit proxies;
in response to a second request to scrape data from a second target server, the second request from a second client associated with the proxy pool index:
(f) retrieving a second cursor of the second client, the second cursor defining a second position of the second client within the ordered plurality of exit proxies and the second cursor being different than the first cursor;
(g) selecting, from the ordered plurality of exit proxies, a second exit proxy at the second position according to the second cursor;
(h) transmitting the second request to the second target server via the selected second exit proxy; and
(i) advancing the second cursor to a second next position in the ordered plurality of exit proxies.