System for creating a space-efficient document categorizer for training and testing of automatic categorization engines
View Patent ↗A method of receiving a network resource address and creating a copy of the network resource and storing the created copy in a database is provided. This creation and storage of a copy of the network resource is performed when there does not exist, in the database, information derived from the resource pointed to by the network address.
1. A method comprising:
receiving a locator of a network resource;
determining if a database already contains stored information derived from the network resource at a previous point in time, effectively freezing the network resource to the previous point in time; and
upon determination that the database does not contain stored information derived from the network resource at the previous point in time, storing information derived from the network resource pointed to by the locator of the network resource, the process of storing comprising:
creating a copy of at least a portion of the network resource pointed to by the locator, and
writing the copy to the database.
2. The method of claim 1 wherein the storing of information further comprises compressing the copy prior to writing the copy.
3. The method of claim 1 wherein the storing the information further comprises generating meta information from the copy.
4. The method of claim 3 wherein the generating meta information involves extracting information from the network resource.
5. The method of claim 3 wherein the generating meta information involves deriving information from the network resource.
6. The method of claim 3 wherein the meta information comprises one or more of a file name, a uniform resource locator, a file format, and a language.
7. The method of claim 3 wherein the storing the information further comprises writing the generated meta information to the database.
8. The method of claim 3 further comprising:
receiving instructions to modify the generated meta information; and
modifying the generated meta information in accordance with the received instructions to generate modified meta information.
9. The method of claim 8 wherein the storing the information further comprises writing the modified meta information to the database.
10. The method of claim 1 wherein the storing the information further comprises modifying references to objects within the information to reflect the new location of referenced objects as being stored in the database.
11. The method of claim 1 further comprising sending the copied information to a generic user agent.
12. The method of claim 1 wherein the network resource comprises one or more world wide web pages.
13. The method of claim 12 wherein the one or more world wide web pages comprises a main frame and one or more sub-frames.
14. The method of claim 13 wherein the main frame and one or more sub-frames are stored as a single file.
15. An apparatus comprising:
a storage medium having stored therein a plurality of programming instructions designed to:
receive a locator of a network resource,
determine if a database already contains stored information derived from the network resource at a previous point in time, effectively freezing the network resource to the previous point in time, and
upon determination that the database does not contain stored information derived from the network resource at the previous point in time, store information derived from the network resource pointed to by the locator of the network resource, the process of storing comprising:
creating a copy of at least a portion of the network resource pointed to by the locator, and
writing the copy to the database; and
a processor coupled to the storage medium to execute the programming instructions.
16. The apparatus of claim 15 wherein the database resides on a separate machine from the processor.
17. The apparatus of claim 15 wherein the storing of information further comprises compressing the copy prior to writing the copy.
18. The apparatus of claim 15 wherein the storing the information further comprises generating meta information from the copy.
19. The apparatus of claim 18 wherein the generating meta information involves extracting information from the network resource.
20. The apparatus of claim 18 wherein the generating meta information involves deriving information from the network resource.
21. The apparatus of claim 18 wherein the meta information comprises one or more of a file name, a uniform resource locator, a file format, and a language.
22. The apparatus of claim 18 wherein storing the information further comprises writing the generated meta information to the database.
23. The apparatus of claim 18 wherein the programming instructions are further designed to:
receive instructions to modify the generated meta information; and
modify the generated meta information in accordance with the received instructions to generate modified meta information.
24. The apparatus of claim 23 wherein storing the information further comprises writing the modified meta information to the database.
25. The apparatus of claim 15 wherein the storing the information further comprises modifying references to objects within the information to reflect the new location of referenced objects as being stored in the database.
26. The apparatus of claim 15 further comprising sending the copied information to a generic user agent.
27. The apparatus of claim 15 wherein the network resource comprises one or more world wide web pages.
28. The apparatus of claim 27 wherein the one or more world wide web pages comprises a main frame and one or more sub-frames.
29. The apparatus of claim 28 wherein the main frame and one or more sub-frames are stored as a single file.