Image processing of webpages
A web detection system processes webpage information and performs automated feature extraction of webpages including machine processable information. In an embodiment, the web detection system determines a subset of webpages having a target characteristic by processing markup language. For a webpage of the subset, the web detection system determines that a first image overlaps at least a portion of a second image in the webpage. The web detection system generates an image of the webpage such that the portion of the second image is obscured by the first image. The web detection system determines a graphical feature of the webpage by processing the image, e.g., using optical character recognition. Responsive to determining that the graphical feature corresponds to graphical features of images of a different set of webpages associated with a target entity, the web detection system determines that the webpage is also associated with the target entity.
1. A method for automated feature extraction of webpages, the method comprising:
determining, by processing markup language of a webpage, that a first object overlaps at least a portion of a second object in the webpage as rendered on a display, comprising:
determining a first set of dimensions of the first object and a second set of dimensions of the second object based on the markup language of the webpage; and
determining that the first object overlaps the second object based on the respective sets of dimensions;
generating an image of the webpage such that the portion of the second object is obscured or altered by the first object;
determining, by processing the image of the webpage, a feature of the webpage corresponding to the portion of the second object;
identifying a match between the feature of the webpage and one or more features of a set of other webpages associated with a target entity; and
generating, responsive to identifying the match, an association between the webpage and the target entity for storage in a database.
2. The method of claim 1 , wherein
the first set of dimensions are of a first image represented by the first object; and
the second set of dimensions are of a second image represented by the second object.
3. The method of claim 2 , wherein determining the feature comprises:
determining a graphical feature of the second image overlapped by the first image, wherein the graphical feature is a logo or color associated with the target entity.
4. The method of claim 1 , further comprising:
determining a first string represented by the portion of the second object;
determining a second string represented by another portion of the first object that overlaps the portion of the second object; and
wherein determining the feature includes determining a deviation between the first string and the second string.
5. The method of claim 4 , wherein the first string describes contact information of the target entity.
6. The method of claim 1 , further comprising:
determining a different feature of the webpage; and
determining that the different feature does not match the one or more features of the set of other webpages associated with the target entity.
7. The method of claim 1 , further comprising:
providing a notification of the association between the webpage and the target entity to a client device for presentation to prompt remedial action.
8. The method of claim 1 , wherein the target entity is a supplier of goods or services, and wherein the set of other webpages associated with the target entity serve as an online storefront for the goods or services.
9. The method of claim 1 , wherein the image of the webpage is generated according to geographical location.
10. The method of claim 9 , wherein the geographical location is a country or defined by a geo-fence.
11. A non-transitory computer-readable storage medium storing instructions for automated feature extraction of webpages, the instructions when executed by a processor causing the processor to:
determine, by processing markup language of a webpage, that a first object overlaps at least a portion of a second object in the webpage as rendered on a display, comprising:
determine a first set of dimensions of the first object and a second set of dimensions of the second object based on the markup language of the webpage; and
determine that the first object overlaps the second object based on the respective sets of dimensions;
generate an image of the webpage such that the portion of the second object is obscured or altered by the first object;
determine, by processing the image of the webpage, a feature of the webpage corresponding to the portion of the second object;
identify a match between the feature of the webpage and one or more features of a set of other webpages associated with a target entity; and
generate, responsive to identifying the match, an association between the webpage and the target entity for storage in a database.
12. The non-transitory computer-readable storage medium of claim 11 , wherein
the first set of dimensions are of a first image represented by the first object; and
the second set of dimensions are of a second image represented by the second object.
13. The non-transitory computer-readable storage medium of claim 12 , wherein determine the feature comprises:
determine a graphical feature of the second image overlapped by the first image, wherein the graphical feature is a logo or color associated with the target entity.
14. The non-transitory computer-readable storage medium of claim 11 , the instructions when executed by the processor further causing the processor to:
determine a first string represented by the portion of the second object;
determine a second string represented by another portion of the first object that overlaps the portion of the second object; and
wherein determine the feature includes determine a deviation between the first string and the second string.
15. The non-transitory computer-readable storage medium of claim 14 , wherein the first string describes contact information of the target entity.
16. The non-transitory computer-readable storage medium of claim 11 , the instructions when executed by the processor further causing the processor to:
determine a different feature of the webpage; and
determine that the different feature does not match the one or more features of the set of other webpages associated with the target entity.
17. The non-transitory computer-readable storage medium of claim 11 , the instructions when executed by the processor further causing the processor to:
provide a notification of the association between the webpage and the target entity to a client device for presentation to prompt remedial action.
18. The non-transitory computer-readable storage medium of claim 11 , wherein the target entity is a supplier of goods or services, and wherein the set of other webpages associated with the target entity serve as an online storefront for the goods or services.
19. The non-transitory computer-readable storage medium of claim 11 , wherein the image of the webpage is generated according to geographical location.
20. The non-transitory computer-readable storage medium of claim 19 , wherein the geographical location is a country or defined by a geo-fence.