Web crawler optimization system
Techniques for optimizing the performance of a webpage crawler are described. According to various embodiments, historical web crawler performance data is accessed, the data describing a performance of a web crawler during various time periods in one or more prior days. A capacity of the web crawler to fulfill uniform resource locator (URL) crawl requests for an upcoming given time period is then estimated, based on the historical web crawler performance data. Thereafter, a plurality of URL crawl requests are distributed to the web crawler during the upcoming given time period, based on the estimated capacity of the web crawler.
1. A computer-implemented method comprising:
accessing historical web crawler performance data describing a performance of a web crawler during various time periods in one or more prior days;
estimating a capacity of the web crawler to fulfill uniform resource locator (URL) crawl requests for an upcoming given time period, the estimating including:
calculating, based on the historical web crawler performance data, a first average of a number of item listing page URL crawl requests completed by the web crawler during the given time period in the one or more prior days;
calculating, based on the historical web crawler performance data, a second average of a number of item listing page URL crawl requests completed by the web crawler during an adjacent time period that is adjacent to the given time period in the one or more prior days; and
calculating a maximum average value based on a maximum of the first average and the second average;
distributing one or more URL crawl requests to the web crawler during the upcoming given time period, based on the estimated capacity of the web crawler;
fetching a plurality of URL crawl requests from one or more data sources, each of the URL crawl requests being associated with use case information;
inserting the fetched URL crawl requests into a request pool;
performing a data deduplication process on the URL crawl requests in the request pool, based on URL crawl requests previously distributed to the web crawler during a recent time period; and
prioritizing the URL crawl requests into a prioritized list of URL crawl requests, based on the use case information associated with each of the URL crawl requests.
2. The method of claim 1 , wherein the historical web crawler performance data identifies a number of URL crawl requests completed by the web crawler during the various time periods in the one or more prior days.
3. The method of claim 1 , wherein the estimating further comprises:
accessing a preconfigured estimate for the daily capacity of the crawler;
assigning a portion of the preconfigured estimate to the given time period;
modifying the portion of the preconfigured estimate based on a daily performance of the crawler during one or more preceding days; and
selecting a maximum of the modified estimate and the maximum average value as the capacity of the web crawler during the given time period.
4. The method of claim 3 , wherein the modifying further comprises:
determining whether the web crawler has fulfilled a threshold value of URL crawl requests during the one or more preceding days;
incrementing the preconfigured estimate by a performance modification factor, responsive to determining that the web crawler has fulfilled a threshold value of URL crawl requests during the one or more preceding days; and
decrementing the preconfigured estimate by a performance modification factor, responsive to determining that the web crawler has not fulfilled the threshold value of URL crawl requests during the one or more preceding days.
5. The method of claim 1 , wherein the data sources include a database specifying URLs associated with popular products posted for sale on a retailer website.
6. The method of claim 1 , wherein the use case information associated with a specific URL crawl requests includes a priority, a frequency, and a service level agreement (SLA) time associated with the specific URL crawl request.
7. The method of claim 1 , wherein the distributing further comprises:
extracting, from the prioritized list of URL crawl requests, an amount of prioritized URL crawl requests corresponding to the estimated capacity of the web crawler; and
transmitting the extracted URL crawl requests to one or more queues associated with the web crawler.
8. The method of claim 7 , wherein the one or more queues include a service level agreement (SLA) queue and a non-SLA queue associated with the web crawler.
9. A system comprising:
one or more processors and executable instructions accessible on a computer-readable medium that, when executed, cause the one or more processors to perform operations comprising:
access historical web crawler performance data describing a performance of a web crawler during various time periods in one or more prior days;
estimate a capacity of the web crawler to fulfill uniform resource locator (URL) crawl requests for an upcoming given time period, including:
calculate, based on the historical web crawler performance data, a first average of a number of item listing page URL crawl requests completed by the web crawler during the given time period in the one or more prior days;
calculate, based on the historical web crawler performance data, a second average of a number of item listing page URL crawl requests completed by the web crawler during an adjacent time period that is adjacent to the given time period in the one or more prior days; and
calculate a maximum average value based on a maximum of the first average and the second average;
distribute one or more URL crawl requests to the web crawler during the upcoming given time period, based on the estimated capacity of the web crawler;
fetch a plurality of URL crawl requests from one or more data sources, each of the URL crawl requests being associated with use case information;
insert the fetched URL crawl requests into a request pool;
perform a data deduplication process on the URL crawl requests in the request pool, based on URL crawl requests previously distributed to the web crawler during a recent time period; and
prioritize the URL crawl requests into a prioritized list of URL crawl requests, based on the use case information associated with each of the URL crawl requests.
10. The system of claim 9 , wherein the capacity analyzer module is further configured to:
access a preconfigured estimate for the daily capacity of the crawler;
assign a portion of the preconfigured estimate to the given time period;
modify the portion of the preconfigured estimate based on a daily performance of the crawler during one or more preceding days; and
select a maximum of the portion of the modified estimate and the maximum average value as the capacity of the web crawler during the given time period.
11. The system of claim 9 , wherein the capacity analyzer module is further configured to:
extract, from the prioritized list of URL crawl requests, an amount of prioritized URL crawl requests corresponding to the estimated capacity of the web crawler; and
transmit the extracted URL crawl requests to one or more queues associated with the web crawler.
12. A hardware storage device having stored therein a set of program instructions which, when executed by a machine causes the machine to perform operations comprising:
accessing historical web crawler performance data describing a performance of a web crawler during various time periods in one or more prior days;
estimating a capacity of the web crawler to fulfill uniform resource locator (URL) crawl requests for an upcoming given time period, the estimating including:
calculating, based on the historical web crawler performance data, a first average of a number of item listing page URL crawl requests completed by the web crawler during the given time period in the one or more prior days;
calculating, based on the historical web crawler performance data, a second average of a number of item listing page URL crawl requests completed by the web crawler during an adjacent time period that is adjacent to the given time period in the one or more prior days; and
calculating a maximum average value based on a maximum of the first average and the second average;
distributing one or more URL crawl requests to the web crawler during the upcoming given time period, based on the estimated capacity of the web crawler;
fetching a plurality of URL crawl requests from one or more data sources, each of the URL crawl requests being associated with use case information;
inserting the fetched URL crawl requests into a request pool;
performing a data deduplication process on the URL crawl requests in the request pool, based on URL crawl requests previously distributed to the web crawler during a recent time period; and
prioritizing the URL crawl requests into a prioritized list of URL crawl requests, based on the use case information associated with each of the URL crawl requests.
13. The hardware storage device of claim 12 , wherein the estimating further comprises:
accessing a preconfigured estimate for the daily capacity of the crawler;
assigning a portion of the preconfigured estimate to the given time period;
modifying the portion of the preconfigured estimate based on a daily performance of the crawler during one or more preceding days; and
selecting a maximum of the modified estimate and the maximum average value as the capacity of the web crawler during the given time period.
14. The hardware storage device of claim 12 , wherein the distributing further comprises:
extracting, from the prioritized list of URL crawl requests, an amount of prioritized URL crawl requests corresponding to the estimated capacity of the web crawler; and
transmitting the extracted URL crawl requests to one or more queues associated with the web crawler.