Method and apparatus for defining data of interest
Some embodiments of the invention include tools for extracting data of interest from the world wide web (WWW). The extraction is accomplished using descriptions of data of interest. The descriptions of data of interest can include computer programs comprising a sequence of instructions and extractor patterns. The extractor patterns can be developed interactively using a web browser integrated into the graphical development environment for creating the descriptions of data of interest. The instructions can be selected from a predetermined list of instructions designed for extracting information from the WWW. The descriptions of data of interest can be grouped into categories sharing common query elements.
1. A computer-implemented method for extracting data from a data source by one or more computing devices, the method comprising:
applying, by at least one of the one or more computing devices, an extractor pattern to a computer-readable data source, wherein the extractor pattern is generated based at least in part on a prior identification of data of interest and wherein the extractor pattern includes one or more regular expressions which are configured to identify additional data of interest in the computer-readable data source;
retrieving, by at least one of the one or more computing devices, the additional data of interest from the computer-readable data source using the one or more regular expressions; and
storing, by at least one of the one or more computing devices, the additional data of interest in a data storage device.
2. The method of claim 1 , wherein applying the extractor pattern to the computer-readable data source comprises applying, by at least one of the one or more computing devices, one or more extraction instructions in conjunction with the extractor pattern, the one or more extraction instructions being selected from a plurality of extraction instructions.
3. The method of claim 1 , wherein the computer-readable data source is part of a web site.
4. The method of claim 1 , wherein the extractor pattern is specific to the computer-readable data source and wherein other extractor patterns are used to extract data from other data sources.
5. The method of claim 1 , wherein the one or more regular expressions include at least one regular expression containing a value received from a user for use as an extraction parameter.
6. The method of claim 1 , wherein at least one of the one or more regular expressions specifies information associated with a product.
7. The method of claim 1 , wherein the one or more regular expressions comprise a pre-condition regular expression which specifies a condition which signals the beginning of potential additional data of interest, a portion-of-data regular expression which defines the additional data of interest, and a post-condition regular expression which specifies a condition which signals the end of the potential additional data of interest.
8. The method of claim 7 , wherein the additional data of interest is extracted from the potential additional data of interest using the portion-of-data regular expression.
9. The method of claim 1 , wherein at least one of the one or more regular expressions specifies a format or syntax condition which is used to identify the additional data of interest.
10. An apparatus for extracting data from a data source, the apparatus comprising:
one or more processors; and
one or more memories operatively coupled to at least one of the one or more processors and having instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:
apply an extractor pattern to a computer-readable data source, wherein the extractor pattern is generated based at least in part on a prior identification of data of interest and wherein the extractor pattern includes one or more regular expressions which are configured to identify additional data of interest in the computer-readable data source;
retrieve the additional data of interest from the computer-readable data source using the one or more regular expressions; and
store the additional data of interest in a data storage device.
11. The apparatus of claim 10 , wherein the instructions that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to apply the extractor pattern to the computer-readable data source further cause at least one of the one or more processors to apply one or more extraction instructions in conjunction with the extractor pattern, the one or more extraction instructions being selected from a plurality of extraction instructions.
12. The apparatus of claim 10 , wherein the computer-readable data source is part of a web site.
13. The apparatus of claim 10 , wherein the extractor pattern is specific to the computer-readable data source and wherein other extractor patterns are used to extract data from other data sources.
14. The apparatus of claim 10 , wherein one or more regular expressions include at least one regular expression containing a value received from a user for use as an extraction parameter.
15. The apparatus of claim 10 , wherein at least one of the one or more regular expressions specifies information associated with a product.
16. The apparatus of claim 10 , wherein the one or more regular expressions comprise a pre-condition regular expression which specifies a condition which signals the beginning of potential additional data of interest, a portion-of-data regular expression which defines the additional data of interest, and a post-condition regular expression which specifies a condition which signals the end of the potential additional data of interest.
17. The apparatus of claim 16 , wherein the additional data of interest is extracted from the potential additional data of interest using the portion-of-data regular expression.
18. The apparatus of claim 10 , wherein at least one of the one or more regular expressions specifies a format or syntax condition which is used to identify the additional data of interest.
19. At least one non-transitory computer-readable medium storing computer-readable instructions that, when executed by one or more computing devices, cause at least one of the one or more computing devices to:
apply an extractor pattern to a computer-readable data source, wherein the extractor pattern is generated based at least in part on a prior identification of data of interest and wherein the extractor pattern includes one or more regular expressions which are configured to identify additional data of interest in the computer-readable data source;
retrieve the additional data of interest from the computer-readable data source using the one or more regular expressions; and
store the additional data of interest in a data storage device.
20. The at least one non-transitory computer-readable medium of claim 19 , wherein the instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to apply the extractor pattern to the computer-readable data source further cause at least one of the one or more computing devices to apply one or more extraction instructions in conjunction with the extractor pattern, the one or more extraction instructions being selected from a plurality of extraction instructions.
21. The at least one non-transitory computer-readable medium of claim 19 , wherein the computer-readable data source is part of a web site.
22. The at least one non-transitory computer-readable medium of claim 19 , wherein the extractor pattern is specific to the computer-readable data source and wherein other extractor patterns are used to extract data from other data sources.
23. The at least one non-transitory computer-readable medium of claim 19 , wherein the one or more regular expressions include at least one regular expression containing a value received from a user for use as an extraction parameter.
24. The at least one non-transitory computer-readable medium of claim 19 , wherein at least one of the one or more regular expressions specifies information associated with a product.
25. The at least one non-transitory computer-readable medium of claim 19 , wherein the one or more regular expressions comprise a pre-condition regular expression which specifies a condition which signals the beginning of potential additional data of interest, a portion-of-data regular expression which defines the additional data of interest, and a post-condition regular expression which specifies a condition which signals the end of the potential additional data of interest.
26. The at least one non-transitory computer-readable medium of claim 25 , wherein the additional data of interest is extracted from the potential additional data of interest using the portion-of-data regular expression.
27. The at least one non-transitory computer-readable medium of claim 19 , wherein at least one of the one or more regular expressions specifies a format or syntax condition which is used to identify the additional data of interest.