System and method for data retrieval and collection in a structured format
View Patent ↗A data collection and retrieval system and associated method allow the capture and replication of data presented at various Web pages into a database application through text parsing of the HTML source code of that document. The system allows the user to select one or more Web pages containing data of interest; to specify exactly which data within any page is to be captured; to specify how frequently data is to be collected; and to specify the conditions for collection and retrieval. The advantage of the system is realized through efficient, automated data collection that would otherwise be impractical. The system includes an initialization stage and an automatic execution stage. The initialization stage provides the user interface which allows the user to select the source file that contains data the user wishes to copy, target database that will receive the data, and timing criteria for automatic transfer of data. The automatic execution stage automatically transfers the data from the source file to the target database as instructed by the user in the initialization stage.
1. A computer-implemented method of selectively retrieving data from a source file on a public network and automatically collecting retrieved data in a target database, comprising:
initializing a data retrieval stage by:
selecting the source file;
parsing selected data elements in the source file into a list of data elements;
selecting at least one of the data elements from the list to be stored in the target database;
identifying source locations of the selected at least one of the data elements;
selecting the target database and target locations in the target database where the at least one of the data elements is to be stored in the target database;
saving the source locations and target locations of the at least one of the parsed data elements in a location file in the target database;
setting a schedule frequency for transferring the at least one of the parsed data elements from the source locations to the target locations in the location file in the target database; and
automatically, selectively parsing new data elements corresponding to the at least one of the parsed data elements previously selected during the initialization of the data retrieval stage, pursuant to the schedule frequency, by transferring the selectively parsed new data elements from the saved source locations in the source file to the saved target locations in the target database.
2. The method of claim 1 , further comprising processing the selectively parsed new data elements stored in the target database.
3. The method of claim 2 , wherein processing the selectively parsed new data elements is executed automatically.
4. The method of claim 1 , wherein the source file is formatted with tags.
5. The method of claim 1 , wherein the at least one of the data elements from the sources locations are derived from an application output file.
6. The method of claim 1 , wherein the at least one of the data elements from the sources locations are derived from an output of a server program.
7. The method of claim 1 , wherein the target database is a relational database.
8. The method of claim 1 , wherein initializing the data retrieval stage further comprises presenting selection options to a user.
9. The method of claim 1 , wherein the at least one of the data elements is dynamic.
10. The method of claim 9 , wherein initializing the data retrieval stage further comprises analyzing the new data elements.
11. The method of claim 10 , wherein the new data elements comprise data that changes periodically.
12. The method of claim 1 , wherein processing the new data elements is executed manually.
13. The method of claim 1 , further comprising automatically updating data elements stored in the target database.
14. The method of claim 13 , wherein automatically updating the data elements stored in the target database comprises periodically retrieving new data elements and updating the data elements stored in the target database with the retrieved new data elements.
15. The method of claim 1 , further comprising reinitializing the data retrieval stage.
16. A computer program product having instruction codes for selectively retrieving data from a source file on a public network and for automatically collecting retrieved data in a target database, comprising:
a first set of instruction codes for initializing a data retrieval stage by:
selecting the source file;
parsing selected data elements in the source file into a list of data elements;
selecting at least one of the data elements from the list to be stored in the target database;
identifying source locations of the selected at least one of the data elements;
selecting the target database and target locations in the target database where the at least one of the data elements is to be stored in the target database;
saving the source locations and target locations of the at least one of the parsed data elements in a location file in the target database;
setting a schedule frequency for transferring the at least one of the parsed data elements from the source locations to the target locations in the location file in the target database; and
a second set of instruction codes for automatically, selectively parsing new data elements corresponding to the at least one of the parsed data elements previously selected during the initialization of the data retrieval stage, pursuant to the schedule frequency, by transferring the selectively parsed new data elements from the saved source locations in the source file to the saved target locations in the target database.
17. The computer program product of claim 16 , wherein the source file is formatted with tags.
18. The computer program product of claim 16 , wherein the at least one of the data elements from the sources locations are derived from an application output file.
19. The computer program product of claim 16 , wherein the at least one of the data elements from the sources locations are derived from an output of a server program.
20. A computer-implemented system for selectively retrieving data from a source file on a public network and for automatically collecting retrieved data in a target database, comprising:
means for initializing a data retrieval stage by:
selecting the source file;
parsing selected data elements in the source file into a list of data elements;
selecting at least one of the data elements from the list to be stored in the target database;
identifying source locations of the selected at least one of the data elements;
selecting the target database and target locations in the target database where the at least one of the data elements is to be stored in the target database;
saving the source locations and target locations of the at least one of the parsed data elements in a location file in the target database;
setting a schedule frequency for transferring the at least one of the parsed data elements from the source locations to the target locations in the location file in the target database; and
means for automatically, selectively parsing new data elements corresponding to the at least one of the parsed data elements previously selected during the initialization of the data retrieval stage, pursuant to the schedule frequency, by transferring the selectively parsed new data elements from the saved source locations in the source file to the saved target locations in the target database.