Extracting data content items using template matching
Systems and methods for extracting data content items from a web page are provided. A template is created by labeling data content items of interest associated with a web page and generating a template Document Object Model (DOM) tree based on the labeled web page. DOM trees are also generated for additional web pages that contain data content items for which extraction may be desired. These DOM trees are compared to the template DOM tree to determine alignment there between. The aligned data content items may then be extracted from the additional web pages and indexed, as desired. Labeling the data content items of interest prior to generating a template DOM tree allows for the desired data content items to be specified and more accurately extracted from related and/or similarly structured web pages.
1. One or more computer storage media having computer-executable instructions embodied thereon for performing a method for extracting data content items from web pages, the method comprising:
receiving a first web page having one or more data content items associated therewith;
receiving an indication to label at least one of the data content items associated with the first web page;
generating, utilizing a computing device, a Document Object Model (DOM) tree associated with the first web page, the DOM tree having a node associated with each data content item;
labeling, utilizing the computing device, the node of the DOM tree associated with the at least one indicated data content item to generate a template DOM tree;
comparing, utilizing the computing device, the template DOM tree with a DOM tree associated with a second web page to determine alignment there between; and
if it is determined that a node of the DOM tree associated with the second web page aligns with the labeled node associated with the template DOM tree, extracting, utilizing the computing device, a data content item from the second web page that is associated with the aligned node of the DOM tree.
2. The computer storage media of claim 1 , wherein the one or more data content items associated with the first web page comprise items associated with an individual or organization data listing.
3. The computer storage media of claim 1 , wherein one or more data content items associated with the first web page comprise one or audio content items, video content items, images, text, or any combination thereof.
4. The computer storage media of claim 1 , wherein at least two of the data content items associated with the template DOM tree have a repeated pattern.
5. The computer storage media of claim 4 , wherein labeling the node of the DOM tree associated with the at least one indicated data content item to create a template DOM tree further comprises labeling a node of the template DOM tree associated with one of the at least two data content items having a repeated pattern with a repeat label.
6. The computer storage media of claim 1 , wherein comparing the template DOM tree with a DOM tree associated with a second web page to determine alignment there between comprises measuring the top-down distance between one node of the template DOM tree and a corresponding node of the DOM tree associated with a second web page.
7. The computer storage media of claim 1 , wherein comparing the template DOM tree with a DOM tree associated with a second web page to determine alignment there between comprises employing partial tree alignment to insert one or more nodes into the template DOM tree.
8. The computer storage media of claim 1 , wherein the method further comprises indexing the extracted data content item.
9. A computer system including at least one computer storage media having computer-executable components embodied thereon for performing a method for extracting data content items from web pages, the computer-executable components comprising:
a receiving component configured for receiving, utilizing a computing device, a plurality of web pages, each web page having at least one data content item associated therewith;
a Document Object Model (DOM) tree generating component configured for generating, utilizing the computing device, a DOM tree associated with one or more of the received web pages;
a labeling component configured for labeling, utilizing the computing device, at least one node associated with a generated DOM tree in accordance with a received labeling indication;
a comparing component configured for comparing, utilizing the computing device, a first DOM tree having at least one labeled node associated therewith a second DOM tree; and
an extracting component configured for extracting, utilizing the computing device, at least one data content item associated with the second DOM tree in accordance with the at least one labeled note associated with the first DOM tree.
10. The computer system of claim 9 , wherein the at least one data content item comprises at least one item associated with an individual or organization listing, an audio content item, a video content item, an image, text, or any combination thereof.
11. The computer system of claim 9 , further comprising a search request receiving component configured for receiving a search request for at least one satisfying data content item.
12. The computer system of claim 11 , further comprising a presenting component configured for presenting the at least one satisfying data content item.
13. The computer system of claim 9 , further comprising an aligning component configured for aligning the matching nodes of the first DOM tree and the second DOM tree.
14. The computer system of claim 9 , wherein the comparing component is further configured for measuring the top-down distance between the first DOM tree and the second DOM tree.
15. The computer system of claim 9 , wherein the comparing component is further configured to utilize partial tree alignment algorithm to insert at least one node into the first DOM tree.
16. A method in a computing environment for extracting data content items from a web page, at least two of the data content items having a repeated pattern, the method comprising:
receiving a first web page having a plurality of data content items associated therewith;
receiving an indication to label at least two of the plurality of data content items, wherein the at least two of the plurality of data content items have a repeated pattern;
generating, utilizing a computing device, a Document Object Model (DOM) tree associated with the first web page, the DOM tree having a node associated with each of the plurality of data content items;
labeling, utilizing the computing device, the nodes of the DOM tree to create a template DOM tree, wherein a node associated with one of the at least two data content items having a repeated pattern is labeled as a repeat node;
comparing, utilizing the computing device, the template DOM tree with a DOM tree associated with a second web page to determine alignment there between; and
if it is determined that a node of the DOM tree associated with the second web page aligns with the labeled node associated with the template DOM tree, extracting, utilizing the computing device, one or more data content items from the second web page that are associated with the aligned repeat node of the DOM tree.
17. The method of claim 16 , wherein the plurality of data content items associated with the first web page comprise items associated with an individual or organization listing.
18. The method of claim 16 , wherein the plurality of data content items associated with the first web page comprise one or audio content items, video content items, images, text, or any combination thereof.
19. The method of claim 16 , wherein comparing the template DOM tree with a DOM tree associated with a second web page comprises measuring the top-down distance between one node of the template DOM tree and a corresponding node of the DOM tree associated with a second web page.
20. The method of claim 16 , wherein comparing the template DOM tree with a DOM tree associated with a second web page comprises using partial tree alignment to insert one or more nodes into the template DOM tree.