AUTOMATIZED PARSING TEMPLATE CUSTOMIZER
Systems and methods to intelligently adapt parsing rules according to the layout changes occurring in multiple targets are disclosed. Specifically, the disclosure provides a solution to detect the layout changes in a target domain and to update parsing templates or parsing rules. The disclosed embodiments in one aspect describe methods and systems to receive and store parsing templates or parsing rules and monitoring tables or a list of related URLs within an internal storage facility. Methods and systems to scrape and parse data by following parsing rules or using parsing templates. The methods and systems describe the manner in which the parsed data and the actual data are analyzed to detect any changes in the layout of the target domain(s). The methods and systems give details on how to decide whether to update parsing rules or parsing templates depending on the layout changes in the target domains.
1 . A method of assessing a web page parsing template comprising:
sending, by a scraping engine, a parsing template and scraped data to parsing engine, wherein the scraped data are associated with the parsing template;
receiving, by the parsing engine, a parsing template and the scraped data from the scraping engine;
parsing, by the parsing engine and using the parsing rules specified in the parsing template, the scraped data to create a parsed data;
adding, by the parsing engine, the parsed data to a monitoring table associated with the parsing template in a storage unit, wherein the monitoring table comprises one or more uniform resource locators (URLs) of a target web domain, and each URL of the one or more URLs is accompanied by actual data of data fields specified in the parsing template;
fetching, by the data analyzer, the actual data and the parsed data from the monitoring table in the storage unit;
calculating, by the data analyzer, an accuracy percentage for the parsing template using the actual data and the parsed data;
comparing, by the data analyzer, the accuracy percentage against an acceptable accuracy level to determine whether to update the parsing template; and
when the accuracy percentage is below the acceptable accuracy level, updating, by a template updater, the parsing template with new parsing rules for the actual data, the new parsing rules selected from a plurality of parsing rules based on a number of occurrences of respective parsing rules from the plurality of parsing rules.
2 . The method of claim 1 , further comprising, when the data analyzer decides to update the parsing template, informing the template updater.
3 . The method of claim 2 , further comprising, removing previously present parsing rules in the parsing template.
4 . The method of claim 3 , wherein the new parsing rules for the actual data comprises parsing rules that are most occurring for each data field.
5 . The method of claim 2 , further comprising, when the accuracy percentage is equal to or higher than the acceptable accuracy level, keeping the parsing template unchanged.
6 . The method of claim 2 , wherein the template updater accesses and fetches the actual data and the scraped data from the storage unit. (Original) The method of claim 1 , wherein the parsed data corresponds to the data fields specified in the parsing template.
8 . The method of claim 1 , wherein the scraped data are stored in the storage unit.
9 . The method of claim 1 , wherein the parsing template and its monitoring table are stored within the storage unit.
10 . The method of claim 1 , wherein the acceptable accuracy level is configured by the service provider infrastructure.
11 . The method of claim 1 , wherein the parsing template comprises parsing rules of the data belonging to the specific data fields.
12 . The method of claim 11 , wherein the parsing rules comprise any of the following or combination thereof: a) XPaths; b) CSS selector strings; c) a combination of XPaths and programming algorithms.
13 . The method of claim 1 , wherein the scraped data is gathered by the scraping engine by accessing the URLs of a target web domain upon request from a client.
14 . (canceled)
15 . (canceled)
16 . The method of claim 1 , wherein the accuracy percentage is a numerical metric for deciding whether the parsing template needs updating and to ascertain the accuracy levels of the parsing template.
17 . The method of claim 1 , wherein the actual data refers to the actual value that corresponds to a specific data field from the URLs of the target web domain.
18 . A method of assessing a web page parsing template comprising:
sending, by a scraping engine, a parsing template and scraped data to parsing engine, wherein the scraped data are associated with the parsing template;
receiving by the parsing engine, a parsing template and the scraped data from the scraping engine;
parsing, by the parsing engine, the scraped data using the parsing rules specified in the parsing template to create a parsed data;
adding by the parsing engine, the parsed data to a monitoring table associated with the parsing template in a storage unit, wherein the monitoring table comprises one or more uniform resource locators (URLs) of a target web domain, and each URL of the one or more URLs is accompanied by actual data of data fields specified in the parsing template;
fetching, by the data analyzer, the actual and parsed data from the monitoring table in the storage unit;
calculating, by the data analyzer, an accuracy percentage for the parsing template using the actual and parsed data;
deciding, by the data analyzer, to update the parsing template based on a comparison of the accuracy percentage against the minimum acceptable accuracy level;
informing, by the data analyzer, the template updater to update the parsing template;
finding, by the template updater, new parsing rules for the actual data from the scraped data, the new parsing rules selected from a plurality of parsing rules based on a number of occurrences of respective parsing rules from the plurality of parsing rules; and
updating by the template updater, the parsing template.
19 . The method of claim 18 , wherein the template updater accesses the storage unit and updates the parsing template with the new parsing rules that are most occurring for each data field.