Data extraction approach for retail crawling engine
A computer system extracts product data from a website and correlates product records from multiple sources to one another as corresponding to the same product. A website is crawled efficiently by rendering webpages using a virtual browser that ignores blacklisted elements, extracts data from objects without rendering, and suppressing retrieval of remote resources. Data is extracted according to engine control statements including a selector and extractor. A website may be crawled repeatedly and changes in extracted data may be detected and flagged. Engine control statements may be automatically changed in response to detecting a change in the configuration of the website. Images of product records may be correlated with one another by first comparing text of the product records and selecting images for comparison based on composition. Images are compared using a machine learning model. Images determined to be similar may be presented to a human for a correlation decision.
1 . A method for web crawling comprising:
retrieving a webpage including instructions for rendering a plurality of elements;
rendering a modified version of the webpage in a virtual browser, the rendering comprising processing each element of a first plurality of elements referenced in the webpage such that one or more resources referenced by an element are suppressed from being rendered in the modified version of the webpage;
traversing the first plurality of elements of the webpage to identify selected elements from the plurality of elements based at least in part on a set of extraction instructions;
applying the set of extraction instructions to the selected elements of the webpage to extract first selected data of the webpage according to the set of extraction instructions, wherein the selected data includes data describing a product;
applying the set of extraction instructions to a second plurality of elements of a prior version of the webpage to extract second selected data of the webpage;
generating, for a plurality of data types, metrics of the plurality of elements of the webpage at the inspection time, the metrics representing the at least a portion of the plurality of elements suppressed from being rendered, the metrics based on characteristics of the plurality of elements and include at least:
a first metric associated with a product description;
a second metric associated with a product price; and
a third metric associated with an image, wherein the third metric is generated at least in part by determining a vector representative of a composition of the image based at least in part on a segment mask associated with a feature represented in the image;
determining, for each of the plurality of data types, differences between the metrics and prior metrics generated for the webpage; and
in response to determining at least one difference exceeding a threshold:
generating a side-by-side comparison including a comparison of at least one of a portion of the product description, the product price, or the image; and
outputting the side-by-side comparison for display on a display device.
2 . The method of claim 1 , wherein rendering the webpage in the virtual browser such that rendering of the at least the portion of the plurality of elements is suppressed comprises:
refraining from retrieving data objects referenced by the at least the portion of the plurality of elements.
3 . The method of claim 2 , wherein identifying data in the webpage describing a product comprises storing references to the data objects in a set of extracted data obtained from the webpage.
4 . The method of claim 1 , wherein rendering the webpage in the virtual browser such that rendering of the at least the portion of the plurality of elements is suppressed comprises:
extracting data from the at least the portion of the plurality of elements without rendering the at least the portion of the plurality of elements.
5 . The method of claim 1 , wherein rendering the webpage in the virtual browser such that rendering of the at least the portion of the plurality of elements is suppressed comprises:
identifying the at least the portion of the plurality of elements as being referenced in a blacklist.
6 . The method of claim 1 , wherein the identifying the data in the webpage comprises:
traversing a document object model (DOM) hierarchy; and
identifying links to one or more product pages as being descendants of a predefined data structure in the DOM hierarchy.
7 . The method of claim 1 , wherein determining, for each of the plurality of data types, differences between the metrics and prior metrics generated for the webpage comprises calculating a similarity score based on the metrics and the prior metrics, wherein the side-by-side comparison is generated in response to the similarity score exceeding the threshold.
8 . The method of claim 1 , further comprising:
selecting a refresh rate to provide tracking over time of at least one of price or available inventory of the product featured in the webpage; and
selecting the inspection time based on a time of the prior version and the refresh rate.
9 . The method of claim 1 , wherein the generating the first metric further comprises:
comparing a first product record and a second product record modified using a normalizing process and using a field-wise comparison.
10 . The method of claim 1 , wherein the generating the third metric further comprises:
evaluating segment masks from a one or more machine learning models used to evaluate the image.
11 . A system comprising:
one or more processing devices; and
one or more memory devices operably coupled to the one or more processing devices, the one or more memory devices storing executable code that, when executed by the one or more processing devices, causes the one or more processing devices to at least:
retrieve a webpage including instructions for rendering a plurality of elements;
render a modified version of the webpage in a virtual browser comprising processing each element of a first plurality of elements referenced in the webpage such that one or more resources referenced by an element are suppressed from being rendered in the modified version of the webpage;
traverse the first plurality of elements of the webpage to identify selected elements from the plurality of elements based at least in part on a set of extraction instructions;
apply the set of extraction instructions to the selected elements of the webpage to extract first selected data of the webpage according to the set of extraction instructions, wherein the selected data includes data describing a product;
apply the set of extraction instructions to a second plurality of elements of a prior version of the webpage to extract second selected data of the webpage;
generate, for a plurality of data types, metrics of the plurality of elements of the webpage at the inspection time, the metrics representing the at least a portion of the plurality of elements refrained from being rendered, the metrics based on characteristics of the plurality of elements and include at least:
a first metric associated with a product description;
a second metric associated with a product price; and
a third metric associated with an image, wherein the third metric is generated at least in part by determining a vector representative of a composition of the image based at least in part on a segment mask associated with a feature represented in the image;
determine, for each of the plurality of data types, differences between the metrics and prior metrics generated for the webpage; and
in response to at least one of the differences exceeding a threshold:
generate a side-by-side comparison including a comparison of at least one of a portion of the product description, the product price, or the image; and
output the side-by-side comparison for display on a display device.
12 . The system of claim 11 , wherein the executable code, when executed by the one or more processing devices, further causes the one or more processing devices to:
refrain from retrieving data objects referenced by the at least the portion of the plurality of elements.
13 . The system of claim 12 , wherein the executable code, when executed by the one or more processing devices, further causes the one or more processing devices to:
identify data in the webpage describing a product by storing references to the data objects in a set of extracted data obtained from the webpage.
14 . The system of claim 11 , wherein the executable code, when executed by the one or more processing devices, further causes the one or more processing devices to:
extract data from the at least the portion of the plurality of elements without rendering the at least the portion of the plurality of elements.
15 . The system of claim 11 , wherein the executable code, when executed by the one or more processing devices, further causes the one or more processing devices to:
identify the at least the portion of the plurality of elements as being referenced in a blacklist.
16 . The system of claim 11 , wherein the executable code, when executed by the one or more processing devices, further causes the one or more processing devices to:
traverse a document object model (DOM) hierarchy; and
identify links to one or more product pages as being descendants of a predefined data structure in the DOM hierarchy.
17 . A system, comprising:
one or more processors; and
a memory coupled to the one or more processors and storing program instructions that, when executed by the one or more processors, cause the one or more processors to at least:
retrieve, at a first time, a webpage including instructions for rendering a first plurality of elements;
render a modified version of the webpage in a virtual browser comprising processing each element of a first plurality of elements referenced in the webpage such that one or more resources referenced by an element are suppressed from being rendered in the modified version of the webpage;
traverse the first plurality of elements of the webpage to identify selected elements from the plurality of elements based at least in part on a set of extraction instructions;
apply the set of extraction instructions to the selected elements of the webpage to extract first selected data of the webpage according to the set of extraction instructions, wherein the selected data includes data;
generate, for a plurality of data types, first metrics of the first plurality of elements of the webpage at the first time, the first metrics representing the at least a portion of the first plurality of elements refrained from being rendered, wherein the first metrics are generated at least in part by determining a first vector representative of a first composition of a first image included in the first plurality of elements based at least in part on a first segment mask associated with a first feature represented in the first image;
retrieve, at a second time, the webpage including the instructions for rendering a second plurality of elements;
generate, for the plurality of data types, second metrics of the second plurality of elements of the webpage at the second time, the second metrics representing the at least a portion of the second plurality of elements refrained from being rendered, wherein the second metrics are generated at least in part by determining a second vector representative of a second composition of a second image included in the second plurality of elements based at least in part on a second segment mask associated with a second feature represented in the second image;
determine, for at least one of the plurality of data types, a difference between the first metrics and the second metrics; and
in response to the difference exceeding a threshold:
generate a side-by-side comparison including a comparison of at least one of a portion of corresponding product descriptions, corresponding product prices, or corresponding images; and
output the side-by-side comparison for display on a display device.
18 . The computing system of claim 17 , wherein the program instructions further cause the one or more processors to at least:
identify data in the webpage describing a product;
select a refresh rate to provide tracking over time of at least one of price or available inventory of the product featured in the webpage; and
select the second time based on the first time and the refresh rate.
19 . The computing system of claim 17 , wherein the program instructions further cause the one or more processors to at least:
determine the threshold as a percentage difference between a data type of the first metrics and the second metrics, the percentage difference indicating at least a change in configuration of the webpage.
20 . The computing system of claim 17 , wherein the plurality of data types include at least one of price, inventory, imagery, descriptive text, or formatting controls.