Method and system for form-filling crawl and associating rich keywords
Techniques are provided for the efficient location, processing, and retrieval of local product information derived from web pages generally locatable through form queries submitted to web pages often referred to as the “deep” or “hidden” web. In an embodiment, information such as product information and dealer-location information is located on a web page form such as a dealer-locator form. After location of a suitable web page form, editorial wrapping is performed to create an automated information extraction process. Using the automated information extractor, deep-web crawling is performed. A grid-based extraction of individual business records is performed, and matching and ingestion are performed in conjunction with a business listing database. Finally, metadata tags are added to entries in the business listing database. Metadata tags also may be added to entries in other databases.
1. A method comprising:
automatically observing user interaction with a particular web form;
producing one or more URL fetch patterns corresponding to the particular web form based, at least in part, on the observed user interaction;
generating a first URL, based, at least in part, on the one or more URL fetch patterns, wherein the first URL includes a first value for a particular parameter associated with the particular web form;
automatically crawling the first URL to obtain a first record;
automatically extracting business information from the first record;
applying one or more tags, based on the business information, to an entry in a business listing database;
generating a second URL, based, at least in part, on the one or more URL fetch patterns, wherein the second URL includes a second value for the particular parameter;
wherein the second value is distinct from the first value;
automatically crawling the second URL to obtain a second record; and
automatically extracting second business information from the second record;
wherein the method is performed by one or more computing devices.
2. The method of claim 1 , further comprising:
automatically comparing said first record to a third record from the business listing database;
based on the comparison, automatically determining whether the first record and the third record both describe an identical business.
3. The method of claim 1 , wherein said producing one or more URL fetch patterns corresponding to the particular web form based, at least in part, on the observed user interaction further comprises:
automatically processing the observed user interaction and one or more candidate fetch URLs that are submitted as a result of the observed user interaction to determine a particular template URL for the particular web form and a set of parameters included in the particular template URL;
wherein the set of parameters includes the particular parameter;
identifying a particular dictionary for the particular parameter;
wherein the first value for the particular parameter is indicated by the particular dictionary.
4. The method of claim 3 , wherein said identifying the particular dictionary further comprises:
receiving, via a control associated with the particular parameter on a graphical user interface, information that identifies the particular dictionary; and
associating the particular dictionary with the particular parameter.
5. The method of claim 1 , wherein the business listing database is constructed using a web crawler.
6. The method of claim 1 , wherein the one or more tags are used to identify results of a search query performed using the business listing database.
7. The method of claim 1 , further comprising modifying, based, at least in part, on the business information extracted from at least the first record, information in one or both of (1) the business listing database, and (2) a search engine database.
8. The method of claim 1 , wherein at least one tag of the one or more tags comprises a product brand name and the one or more tags describe (1) products for sale, or (2) services performed by a business having the entry in the business listing database.
9. The method of claim 1 , further comprising:
receiving information identifying a set of one or more values, for the particular parameter, to be included in generated URLs;
wherein the first and second values are included in the set of one or more values.
10. A computer-implemented method comprising:
automatically populating one or more fields of a form on a first web page that is used to locate stores that carry products that belong to a particular brand;
receiving, in response to a submission of said form, one or more records that indicate one or more geographical locations of stores that each carry products that belong to said particular brand;
locating, within a business listing repository, and based on first contact information that is contained in a first record of said one or more records, a second record that contains second contact information that matches said first contact information;
wherein said second contact information was extracted from a second web page by an automated web crawler that automatically follows hyperlinks between web pages;
storing data that associates said particular brand with said second record;
receiving a query that indicates both (a) a particular geographical location and (b) said particular brand;
in response to receiving said query, determining, based at least in part on said data, that said second record is associated with both (a) said particular geographical location and (b) said particular brand; and
in response to determining that said second record is associated with both (a) said particular geographical location and (b) said particular brand, generating a search results web page that contains at least a portion of said second contact information from said second record;
wherein the method is performed by one or more computing devices.
11. One or more non-transitory computer readable media storing instructions which, when executed by one or more computing devices, cause:
automatically populating one or more fields of a form on a first web page that is used to locate stores that carry products that belong to a particular brand;
receiving, in response to a submission of said form, one or more records that indicate one or more geographical locations of stores that each carry products that belong to said particular brand;
locating, within a business listing repository, and based on first contact information that is contained in a first record of said one or more records, a second record that contains second contact information that matches said first contact information;
wherein said second contact information was extracted from a second web page by an automated web crawler that automatically follows hyperlinks between web pages;
storing data that associates said particular brand with said second record;
receiving a query that indicates both (a) a particular geographical location and (b) said particular brand;
in response to receiving said query, determining, based at least in part on said data, that said second record is associated with both (a) said particular geographical location and (b) said particular brand; and
in response to determining that said second record is associated with both (a) said particular geographical location and (b) said particular brand, generating a search results web page that contains at least a portion of said second contact information from said second record.
12. One or more non-transitory computer readable media storing instructions which, when executed by one or more computing devices, cause:
automatically observing user interaction with a particular web form;
producing one or more URL fetch patterns corresponding to the particular web form based, at least in part, on the observed user interaction;
generating a first URL, based, at least in part, on the one or more URL fetch patterns, wherein the first URL includes a first value for a particular parameter associated with the particular web form;
automatically crawling the first URL to obtain a first record;
automatically extracting business information from the first record;
applying one or more tags, based on the business information, to an entry in a business listing database;
generating a second URL, based, at least in part, on the one or more URL fetch patterns, wherein the second URL includes a second value for the particular parameter;
wherein the second value is distinct from the first value;
automatically crawling the second URL to obtain a second record; and
automatically extracting second business information from the second record.
13. The one or more non-transitory computer readable media of claim 12 , further comprising instructions for:
automatically comparing said first record to a third record from the business listing database;
based on the comparison, automatically determining whether the first record and the third record both describe an identical business.
14. The one or more non-transitory computer readable media of claim 12 , wherein said producing one or more URL fetch patterns corresponding to the particular web form based, at least in part, on the observed user interaction further comprises:
automatically processing the observed user interaction and one or more candidate fetch URLs that are submitted as a result of the observed user interaction to determine a particular template URL for the particular web form and a set of parameters included in the particular template URL;
wherein the set of parameters includes the particular parameter;
identifying a particular dictionary for the particular parameter;
wherein the first value for the particular parameter is indicated by the particular dictionary.
15. The one or more non-transitory computer readable media of claim 14 , wherein said identifying the particular dictionary further comprises:
receiving, via a control associated with the particular parameter on a graphical user interface, information that identifies the particular dictionary; and
associating the particular dictionary with the particular parameter.
16. The one or more non-transitory computer readable media of claim 12 , wherein the business listing database is constructed using a web crawler.
17. The one or more non-transitory computer readable media of claim 12 , wherein the one or more tags are used to identify results of a search query performed using the business listing database.
18. The one or more non-transitory computer readable media of claim 12 , further comprising instructions for modifying, based, at least in part, on the business information extracted from at least the first record, information in one or both of (1) the business listing database, and (2) a search engine database.
19. The one or more non-transitory computer readable media of claim 12 , wherein at least one tag of the one or more tags comprises a product brand name and the one or more tags describe (1) products for sale, or (2) services performed by a business having the entry in the business listing database.
20. The one or more non-transitory computer readable media of claim 12 , further comprising instructions for:
receiving information identifying a set of one or more values, for the particular parameter, to be included in generated URLs;
wherein the first and second values are included in the set of one or more values.