Method and apparatus for defining data of interest
View Patent ↗Some embodiments of the invention include tools for extracting data of interest from the world wide web (WWW). The extraction is accomplished using descriptions of data of interest. The descriptions of data of interest can include computer programs comprising a sequence of instructions and extractor patterns. The extractor patterns can be developed interactively using a web browser integrated into the graphical development environment for creating the descriptions of data of interest. The instructions can be selected from a predetermined list of instructions designed for extracting information from the WWW. The descriptions of data of interest can be grouped into categories sharing common query elements. Multiple descriptions of data of interest in the same category can executed simultaneously using the same query. The descriptions of data of interest can be accessed by a client computer using a web browser to initiate a query. In some embodiments, the descriptions of data of interest are used to provide information about products available for sale over the WWW.
1. A computer-implemented method executed by one or more computing devices for extracting data of interest to a user from a web site, the method comprising:
receiving, by at least one of the one or more computing devices, a description of data of interest from a user, the description of the data of interest being associated with an extraction parameter;
querying, by at least one of the one or more computing devices, a web site using a value of the extraction parameter and an extraction pattern, the extraction pattern being associated with the description of data of interest, wherein the extraction pattern is adapted to identify at least a portion of an output of a web site and extract information from one or more web pages associated with the web site, and wherein the extraction pattern comprises a regular expression;
extracting, by at least one of the one or more computing devices, the data of interest from the web site based on the query; and
storing, by at least one of the one or more computing devices, the extracted data of interest.
2. The method of claim 1 , wherein the value of the extraction parameter is received from a user.
3. The method of claim 1 , wherein the user provides the description of the data of interest using a graphical user interface tool.
4. The method of claim 3 , wherein the graphical user interface tool includes a plurality of descriptions of data of interest associated with a plurality of extraction parameters.
5. The method of claim 1 , further comprising:
obtaining additional data of interest by querying another web site using the value of the extraction parameter and the extraction pattern;
extracting the additional data of interest from the other web site; and
storing the additional data of interest.
6. The method of claim 5 , wherein the data of interest is provided incrementally as it is obtained from the web sites.
7. The method of claim 5 , wherein the data of interest is obtained from the web sites and then presented simultaneously.
8. The method of claim 5 , wherein when the data of interest includes data of interest from both web sites, and the data of interest from both web sites is extracted.
9. The method of claim 1 , wherein the data of interest is information associated with a product or information associated with a service.
10. A system for extracting data of interest to a user from a web site, the system comprising:
a processor; and
memory operatively coupled to the processor and containing instructions that, when executed by the processor, cause the processor to carry out the steps of:
receiving a description of data of interest from a user, the description of the data of interest being associated with an extraction parameter;
querying a web site using a value of the extraction parameter and an extraction pattern, the extraction pattern being associated with the description of data of interest, wherein the extraction pattern is adapted to identify at least a portion of an output of a web site and extract information from one or more web pages associated with the web site, and wherein the extraction pattern comprises a regular expression;
extracting the data of interest from the web site based on the query; and
storing the extracted data of interest.
11. The system of claim 10 , wherein the value of the extraction parameter is received from a user.
12. The system of claim 10 , wherein the user provides the description of the data of interest using a graphical user interface tool.
13. The system of claim 12 , wherein the graphical user interface tool includes a plurality of descriptions of data of interest associated with a plurality of extraction parameters.
14. The system of claim 10 , wherein the instructions, when executed by the processor, further cause the processor to carry out the steps of:
obtaining additional data of interest by querying another web site using the value of the extraction parameter and the extraction pattern;
extracting the additional data of interest from the other web site; and
storing the additional data of interest.
15. The system of claim 14 , wherein the data of interest is provided incrementally as it is obtained from the web sites.
16. The system of claim 14 , wherein the data of interest is obtained from the web sites and then presented simultaneously.
17. The system of claim 14 , wherein when the data of interest includes data of interest from both web sites, and the data of interest from both web sites is extracted.
18. The system of claim 10 , wherein the data of interest is information associated with a product or information associated with a service.
19. Non-transitory computer readable media having instructions recorded thereon that, when executed by a processor, cause the processor to carry out a method for extracting data of interest to a user from a web site, the method comprising the steps of:
receiving a description of data of interest from a user, the description of the data of interest being associated with an extraction parameter;
querying a web site using a value of the extraction parameter and an extraction pattern, the extraction pattern being associated with the description of data of interest, wherein the extraction pattern is adapted to identify at least a portion of an output of a web site and extract information from one or more web pages associated with the web site, and wherein the extraction pattern comprises a regular expression;
extracting the data of interest from the web site based on the query; and
storing the extracted data of interest.
20. The computer-readable media of claim 19 , wherein the value of the extraction parameter is received from a user.
21. The computer-readable media of claim 19 , wherein the user provides the description of the data of interest using a graphical user interface tool.
22. The computer-readable media of claim 21 , wherein the graphical user interface tool includes a plurality of descriptions of data of interest associated with a plurality of extraction parameters.
23. The computer-readable media of claim 19 , wherein the instructions, when executed by the processor, further cause the processor to carry out the steps of:
obtaining additional data of interest by querying another web site using the value of the extraction parameter and the extraction pattern;
extracting the additional data of interest from the other web site; and
storing the additional data of interest.
24. The computer-readable media of claim 23 , wherein the data of interest is provided incrementally as it is obtained from the web sites.
25. The computer-readable media of claim 23 , wherein the data of interest is obtained from the web sites and then presented simultaneously.
26. The computer-readable media of claim 23 , wherein when the data of interest includes data of interest from both web sites, and the data of interest from both web sites is extracted.
27. The computer-readable media of claim 19 , wherein the data of interest is information associated with a product or information associated with a service.