Streaming proxy service
Embodiments disclose a system that allows for improved generation of web requests for scraping that, because of the nature of the requests and time and manner they are sent out, appear more organic, as in human generated, than conventional automated scraping systems. The system then manages how a client request to scrape a target website is made to the site, masking the request in a manner that makes it appear to the Web server as if the request is not generated by an automated system. In this way, by appearing more organic, Web servers may be less likely to block requests from the disclosed system or may take longer to block requests from the disclosed system. By avoiding Web servers blocking requests and extending the lifetime of IP proxies before they are blocked, embodiments can use a limited IP proxy address space more efficiently.
1 . A method comprising:
(a) receiving, at a back-end proxy server, a proxy protocol request to a target streaming service, the proxy protocol request including an identifier that uniquely identifies a client of a virtual private network service provider, the client having requested streaming data;
(b) at the back-end proxy server, determining whether the identifier is entered in a table mapping respective identifiers to one of a plurality of internet protocol (IP) addresses;
when the identifier is not entered in the table:
(c) selecting, by the back-end proxy server, an IP address from the plurality of IP addresses by determining the target streaming service is configured to provide the requested streaming data in response to requests from a geographic region of the selected IP address;
(d) adding, by the back-end proxy server, an entry to the table mapping the identifier to the IP address in (c) such that subsequent requests from the client are mapped to the same IP address selected in (c); and
(e) sending, by the back-end proxy server, a message to the target streaming service, wherein the message uses the IP address selected in (c) as the message's source address.
2 . The method of claim 1 , wherein the message is a first message, further comprising, when the identifier is entered in the table:
(f) selecting, by the back-end proxy server, the IP address mapped to the identifier in the table; and
(g) sending, by the back-end proxy server, a second message to the target streaming service, wherein the second message uses the IP address selected in (f).
3 . The method of claim 1 , wherein the selecting (c) comprises:
determining, by the back-end proxy server, the selected IP address is not blocked by the target streaming service.
4 . The method of claim 1 , further comprising:
periodically determining, by the back-end proxy server, whether the selected IP address is blocked by the target streaming service; and
periodically determining, by the back-end proxy server, regional content provided by the target streaming service based on the selected IP address.
5 . The method of claim 1 , further comprising:
receiving, at the back-end proxy server, the requested streaming data from the target streaming service; and
transmitting, by the back-end proxy server, the received streaming data to a front-end proxy server, the front-end proxy server forwarding the received streaming data to the client.
6 . The method of claim 1 , wherein prior to the sending (e), the method further comprises:
determining, by the back-end proxy server, the proxy protocol request is plain text;
extracting, by the back-end proxy server, a domain name from the proxy protocol request; and
resolving, by the back-end proxy server, the extracted domain name via a domain name system.
7 . The method of claim 1 , wherein prior to the sending (e), the method further comprises:
determining, by the back-end proxy server, the proxy protocol request is encrypted;
extracting, by the back-end proxy server, a TLS SNI header from the proxy protocol request; and
resolving, by the back-end proxy server, the extracted TLS SNI header via a domain name system.
8 . The method of claim 1 , wherein the identifier includes an indication of an IP address of the client and a server hostname of the virtual private network service provider.
9 . The method of claim 1 , wherein the selected IP address is different from an IP address of the client.
10 . The method of claim 1 , wherein the IP address is selected based at least on a geographic region associated with the selected IP address, a geographic region of the target streaming service, and the requested streaming data.
11 . A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations, comprising:
(a) receiving a proxy protocol request to a target streaming service, the proxy protocol request including an identifier that uniquely identifies a client of a virtual private network service provider, the client having requested streaming data;
(b) determining whether the identifier is entered in a table mapping respective identifiers to one of a plurality of internet protocol (IP) addresses;
when the identifier is not entered in the table:
(c) selecting an IP address from the plurality of IP addresses based on determining the target streaming service is configured to provide the requested streaming data in response to requests from a geographic region of the selected IP address;
(d) adding an entry to the table mapping the identifier to the IP address in (c) such that subsequent requests from the client are mapped to the same IP address selected in (c); and
(e) sending a message to the target streaming service, wherein the message uses the IP address selected in (c) as the message's source address.
12 . The non-transitory computer-readable device of claim 11 , wherein the message is a first message, the operations further comprise, when the identifier is entered in the table:
(f) selecting the IP address mapped to the identifier in the table; and
(g) sending a second message to the target streaming service, wherein the second message uses the IP address selected in (f).
13 . The non-transitory computer-readable device of claim 11 , wherein the selecting (c) comprises:
determining, by an back-end proxy server, the selected IP address is not blocked by the target streaming service.
14 . The non-transitory computer-readable device of claim 11 , the operations further comprise:
periodically determining whether the selected IP address is blocked by the target streaming service; and
periodically determining regional content provided by the target streaming service based on the selected IP address.
15 . The non-transitory computer-readable device of claim 11 , the operations further comprise:
receiving the requested streaming data from the target streaming service; and
transmitting the received streaming data to a front-end proxy server, the front-end proxy server forwarding the received streaming data to the client.
16 . The non-transitory computer-readable device of claim 11 , wherein prior to the sending (e), the operations further comprise:
determining the proxy protocol request is plain text;
extracting a domain name from the proxy protocol request; and
resolving the extracted domain name via a domain name system.
17 . The non-transitory computer-readable device of claim 11 , wherein prior to the sending (e), the operations further comprise:
determining the proxy protocol request is encrypted;
extracting a TLS SNI header from the proxy protocol request; and
resolving the extracted TLS SNI header via a domain name system.
18 . The non-transitory computer-readable device of claim 11 , wherein the identifier includes an indication of an IP address of the client and a server hostname of the virtual private network service provider.
19 . The non-transitory computer-readable device of claim 11 , wherein the selected IP address is different from an IP address of the client.
20 . The non-transitory computer-readable device of claim 11 , wherein the IP address is selected based at least on a geographic region associated with the selected IP address, a geographic region of the target streaming service, and the requested streaming data.