Utilizing Database Management and Throttling During Proxy Web Scraping
Embodiments disclose a system that allows for improved generation of web requests for scraping that, because of the nature of the requests and time and manner they are sent out, appear more organic, as in human generated, than conventional automated scraping systems. The system then manages how a client request to scrape a target website is made to the site, masking the request in a manner that makes it appear to the Web server as if the request is not generated by an automated system. In this way, by appearing more organic, Web servers may be less likely to block requests from the disclosed system or may take longer to block requests from the disclosed system. By avoiding Web servers blocking requests and extending the lifetime of IP proxies before they are blocked, embodiments can use a limited IP proxy address space more efficiently.
1 .- 20 . (canceled)
21 . A computer implemented method comprising:
(a) during a first time period, receiving, from a client computing device, a web scraping request specifying a target website to capture content from;
(b) sending a job description of the web scraping request to a job database at a database server for storage, wherein the job database stores the status of current web scraping jobs being executed;
(c) determining, based on the job description, a second time period to wait prior to executing the web scraping request, wherein the second time period is longer than the first time period;
(d) after the second time period has passed, generating a web request for the target website as specified in the job description; and
(e) transmitting the web request such that the web request reaches the target website via a proxy selected from a group of proxies.
22 . The computer implemented method of claim 21 , further comprising:
(f) receiving content from the target website according to the web scraping request; and
(g) transmitting the content to the client computing device.
23 . The computer implemented method of claim 21 , wherein the job description is stored within a queue of a plurality of queues at the job database.
24 . The computer implemented method of claim 23 , wherein each queue of the plurality of queues has a unique time period indicating how long to wait prior to generating the web request for the target website.
25 . The computer implemented method of claim 21 , wherein the database server is selected by:
repeatedly checking health of the database server; and
determining, based on a table, the database server is enabled.
26 . The computer implemented method of claim 21 , wherein the database server is one of a plurality of database servers.
27 . The computer implemented method of claim 26 , wherein each of the plurality of database servers is a shard managing storage in a horizontal partition of the job database.
28 . The computer implemented method of claim 21 , wherein the second time period is based on at least one of a level of service of the client computing device, whether the job description specifies that the web scraping request be made from a headless browser, or an identifier of the target website.
29 . The computer implemented method of claim 21 , further comprising:
(f) determining the database server is disabled based on at least one of a nonresponsive connection, an error message, or a number of job descriptions in the job database;
(g) indicating the database server is disabled in the table; and
(h) sending, based on the table, a second job description of a second web scraping request to a second database server.
30 . A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations, comprising:
(a) during a first time period, receiving, from a client computing device, a web scraping request specifying a target website to capture content from;
(b) sending a job description of the web scraping request to a job database at a database server,
wherein the database server is selected for storage of the job description based on performing a health check on the database server,
wherein the job database is configured to store the status of current web scraping jobs being executed,
wherein the database server is one of a plurality of database servers operating independently of one another, and
wherein each of the database servers is configured to manage data storage to at least a portion of the job database,
(c) determining, based on the job description, a second time period to wait prior to executing the web scraping request, wherein the second time period is longer than the first time period;
(d) after the second time period has passed, generating a web request for the target website as specified in the job description; and
(e) transmitting the web request such that the web request reaches the target website via a proxy selected from a group of proxies.
31 . The non-transitory computer-readable device of claim 30 , wherein the operations further comprise:
(f) receiving content from the target website according to the web scraping request; and
(g) transmitting the content to the client computing device.
32 . The non-transitory computer-readable device of claim 30 , wherein the job description is stored within a queue of a plurality of queues at the job database.
33 . The non-transitory computer-readable device of claim 32 , wherein each queue of the plurality of queues has a unique time period indicating how long to wait prior to generating the web request for the target website.
34 . The non-transitory computer-readable device of claim 30 , wherein to perform the health check on the database server, the operations comprise:
transmitting a first message to the database server;
receiving a second message from the database server; and
updating a table indicating the database server is enabled.
35 . The non-transitory computer-readable device of claim 30 , wherein each of the plurality of database servers is a shard managing storage in a horizontal partition of the job database.
36 . The non-transitory computer-readable device of claim 30 , wherein the second time period is based on at least one of a level of service of the client computing device, whether the job description specifies that the web scraping request be made from a headless browser, or an identifier of the target website.
37 . The non-transitory computer-readable device of claim 30 , wherein the operations further comprise:
(f) determining the database server is disabled based on at least one of a nonresponsive connection, an error message, or a number of job descriptions in the job database;
(g) indicating the database server is disabled in the table; and
(h) sending, based on the table, a second job description of a second web scraping request to a second database server.
38 . A system comprising:
at least one processor;
a memory configured to store a queue and a job database storing the status of web scraping jobs while the web scraping jobs are being executed;
a plurality of database servers operating independently of one another, each of the database servers configured to manage data storage to at least a portion of the job database;
a request intake manager configured to, during a first time period, receive a web scraping request from a client computing device, and send a job description of the web scraping request to a selected database server of the plurality of database servers as a pending web scraping job, wherein the web scraping request specifies a target website to capture content from;
a throttler configured to place the job description of the web scraping request on the queue;
a job remover configured to, incrementally over a second time period longer than the first time period, remove the job description from the queue; and
a scraper configured to, when the job description is removed from the queue, generate a web request for the target website as specified in the job description, and transmit the web request such that the web request reaches the target website via a proxy selected from a group of proxies.
39 . The system of claim 38 , wherein the request intake manager is further configured to transmit the content from the target website to the client computing device.
40 . The system of claim 38 , wherein the second period of time is based at least on a level of service of the client computing device, whether the job description specifies that the web scraping request be made from a headless browser, or an identifier of the target website.